Image processing device, information storage device, and image processing method

ABSTRACT

An image processing device including an image sequence acquisition section that acquires an image sequence that includes a plurality of images; and a processing section that performs an image summarization process that deletes some of the plurality of images included in the image sequence acquired by the image sequence acquisition section to acquire a summary image sequence. The processing section detecting a scene change from the image sequence, setting a partial image sequence that includes images among the plurality of images included in the image sequence based on the detected scene change, selecting a reference image and a determination target image from the partial image sequence, and determining whether or not the determination target image can be deleted based on deformation information about the reference image and the determination target image.

CROSS REFERENCE TO RELATED APPLICATION

This application is a Divisional of U.S. Ser. No. 14/514,062, filed Oct.14, 2014, which is a Continuation of International Patent ApplicationNo. PCT/JP2013/058489, having an international filing date of Mar. 25,2013, which designated the United States, the entirety of both of whichare incorporated herein by reference. Japanese Patent Application No.2012-094683 filed on Apr. 18, 2012, Japanese Patent Application No.2012-094691, filed on Apr. 18, 2012 and Japanese Patent Application No.2012-117318 filed on May 23, 2012 are also incorporated herein byreference in their entirety.

BACKGROUND

The present invention relates to an image processing device, a program,an image processing method, and the like.

When still images are continuously captured in time series at given timeintervals, or when a spatial object is covered by a number of images, orwhen a movie is captured, and each image that forms the movie is used asa still image, for example, a very large number of temporally orspatially continuous images (hereinafter may be referred to as “imagesequence”) are acquired. In such a case, it is likely that the imagesthat are closely situated in the image sequence (i.e., images that areclose to each other temporally or spatially) are similar images, and itis not likely that it is necessary to check all of a large number ofimages in order to determine the captured information. Since the numberof images may be tens of thousands or more, it takes time for the userto check all of the images.

Therefore, it has been desired to summarize the original image sequenceusing an image sequence that includes a smaller number of images bydeleting images from the original image sequence. This process ishereinafter referred to as “image summarization process”. For example,JP-A-2009-5020 discloses an image summarization method that extracts ascene change boundary image included in the image sequence, or an imagethat represents the image sequence, and allows images from which theinformation represented by the image sequence can be easily determinedto remain.

When applying an image summarization technique in the medical field, forexample, it is necessary to prevent a situation in which an area thatcannot be observed occurs due to deletion of an image in order to ensurethat a disease can be reliably observed. In particular, it is necessaryto ensure that an important area such as a lesion area or an abnormalarea can be reliably observed.

SUMMARY

According to one aspect of the invention, there is provided an imageprocessing device comprising: an image sequence acquisition section thatacquires an image sequence that includes a plurality of images; and aprocessing section that performs an image summarization process thatdeletes some of the plurality of images included in the image sequenceacquired by the image sequence acquisition section to acquire a summaryimage sequence, the processing section selecting a reference image and adetermination target image from the plurality of images, and determiningwhether or not the determination target image can be deleted based onresults of a process that utilizes deformation information about thereference image and the determination target image, and a process thatutilizes a structural element that corresponds to an attention area.

According to another aspect of the invention, there is provided an imageprocessing device comprising: an image sequence acquisition section thatacquires an image sequence that includes a plurality of images; and aprocessing section that performs an image summarization process thatdeletes some of the plurality of images included in the image sequenceacquired by the image sequence acquisition section to acquire a summaryimage sequence, the processing section selecting a reference image and adetermination target image from the plurality of images, calculating acoverage area that is an area in which the determination target image iscovered by the reference image based on deformation information aboutthe reference image and the determination target image, and determininga probability that an attention area is missed based on the coveragearea to determine whether or not the determination target image can bedeleted.

According to another aspect of the invention, there is provided an imageprocessing device comprising: an image sequence acquisition section thatacquires an image sequence that includes a plurality of images; and aprocessing section that performs an image summarization process thatdeletes some of the plurality of images included in the image sequenceacquired by the image sequence acquisition section to acquire a summaryimage sequence, the processing section detecting a scene change from theimage sequence, setting a partial image sequence that includes imagesamong the plurality of images included in the image sequence based onthe detected scene change, selecting a reference image and adetermination target image from the partial image sequence, anddetermining whether or not the determination target image can be deletedbased on deformation information about the reference image and thedetermination target image.

According to another aspect of the invention, there is provided acomputer-readable storage device with an executable program storedthereon, wherein the program instructs a computer to function as:

an image sequence acquisition section that acquires an image sequencethat includes a plurality of images; and a processing section thatperforms an image summarization process that deletes some of theplurality of images included in the image sequence acquired by the imagesequence acquisition section to acquire a summary image sequence, theprocessing section selecting a reference image and a determinationtarget image from the plurality of images, and determining whether ornot the determination target image can be deleted based on results of aprocess that utilizes deformation information about the reference imageand the determination target image, and a process that utilizes astructural element that corresponds to an attention area.

According to another aspect of the invention, there is provided acomputer-readable storage device with an executable program storedthereon, wherein the program instructs a computer to function as: animage sequence acquisition section that acquires an image sequence thatincludes a plurality of images; and a processing section that performsan image summarization process that deletes some of the plurality ofimages included in the image sequence acquired by the image sequenceacquisition section to acquire a summary image sequence, the processingsection selecting a reference image and a determination target imagefrom the plurality of images, calculating a coverage area that is anarea in which the determination target image is covered by the referenceimage based on deformation information about the reference image and thedetermination target image, and determining a probability that anattention area is missed based on the coverage area to determine whetheror not the determination target image can be deleted.

According to another aspect of the invention, there is provided acomputer-readable storage device with an executable program storedthereon, wherein the program instructs a computer to function as:

an image sequence acquisition section that acquires an image sequencethat includes a plurality of images; and

a processing section that performs an image summarization process thatdeletes some of the plurality of images included in the image sequenceacquired by the image sequence acquisition section to acquire a summaryimage sequence,

the processing section detecting a scene change from the image sequence,setting a partial image sequence that includes images among theplurality of images included in the image sequence based on the detectedscene change, selecting a reference image and a determination targetimage from the partial image sequence, and determining whether or notthe determination target image can be deleted based on deformationinformation about the reference image and the determination targetimage.

According to another aspect of the invention, there is provided an imageprocessing method comprising: acquiring an image sequence that includesa plurality of images; selecting a reference image and a determinationtarget image from the plurality of images included in the imagesequence; determining whether or not the determination target image canbe deleted based on results of a process that utilizes deformationinformation about the reference image and the determination targetimage, and a process that utilizes a structural element that correspondsto an attention area; and performing an image summarization process thatdeletes some of the plurality of images included in the image sequencebased on a result of the determination as to whether or not thedetermination target image can be deleted to acquire a summary imagesequence.

According to another aspect of the invention, there is provided an imageprocessing method comprising: acquiring an image sequence that includesa plurality of images; selecting a reference image and a determinationtarget image from the plurality of images included in the imagesequence; calculating a coverage area based on deformation informationabout the reference image and the determination target image, thecoverage area being an area in which the determination target image iscovered by the reference image; determining a probability that anattention area is missed based on the coverage area to determine whetheror not the determination target image can be deleted; and performing animage summarization process that deletes some of the plurality of imagesincluded in the image sequence based on a result of the determination asto whether or not the determination target image can be deleted toacquire a summary image sequence.

According to another aspect of the invention, there is provided an imageprocessing method comprising: acquiring an image sequence that includesa plurality of images; detecting a scene change from the acquired imagesequence; setting a partial image sequence that includes images amongthe plurality of images included in the image sequence based on thedetected scene change; selecting a reference image and a determinationtarget image from the partial image sequence; and performing an imagesummarization process that determines whether or not the determinationtarget image can be deleted based on deformation information about thereference image and the determination target image, and deletes some ofthe plurality of images included in the image sequence to acquire asummary image sequence.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example of calculation of a coverage area based ondeformation information.

FIGS. 2A to 2C are views illustrating the relationship between the shapeof a non-coverage area and the probability that an attention area ismissed.

FIG. 3 illustrates a system configuration example of an image processingdevice according to one embodiment of the invention.

FIG. 4 illustrates a system configuration example of an image processingdevice according to a first embodiment.

FIG. 5 is a flowchart illustrating a process according to the firstembodiment.

FIGS. 6A to 6E are views illustrating an erosion process that utilizes astructural element on a non-coverage area.

FIGS. 7A to 7D are views illustrating a reference image/determinationtarget image selection process according to the first embodiment.

FIG. 8 illustrates a system configuration example of an image processingdevice according to a second embodiment.

FIG. 9 is a flowchart illustrating a process according to the secondembodiment.

FIG. 10 is a view illustrating a process that calculates a coverage areabased on a plurality of reference images.

FIGS. 11A and 11B are views illustrating a reference image/determinationtarget image selection process according to the second embodiment.

FIGS. 12A to 12D are views illustrating an example of a first referenceimage selection method according to the second embodiment.

FIGS. 13A to 13G are views illustrating a modification of the secondembodiment.

FIG. 14 is a flowchart illustrating a process according to amodification of the second embodiment.

FIG. 15 illustrates a system configuration example of an imageprocessing device according to a third embodiment.

FIGS. 16A to 16C are views illustrating an erosion process that utilizesa structural element on a determination target image.

FIGS. 17A and 17B are views illustrating an inclusion determinationprocess on a reference image and a coverage-requiring area.

FIG. 18 is a flowchart illustrating a process according to the thirdembodiment.

FIGS. 19A to 19C are views illustrating a process that utilizes astructural element according to a fourth embodiment.

FIG. 20 illustrates a system configuration example of an imageprocessing device according to a fifth embodiment.

FIG. 21 illustrates another example of a coverage ratio calculationmethod.

FIG. 22 illustrates a configuration example of an image processingdevice according to a sixth embodiment.

FIG. 23 is a flowchart illustrating a process according to the sixthembodiment.

FIG. 24 is a view illustrating an observation area/corresponding areasetting method.

FIG. 25 illustrates an example in which brightness information is usedas a feature quantity.

FIG. 26 illustrates an example in which size information is used as afeature quantity.

FIG. 27 illustrates an example in which the similarity with a givenshape is used as a feature quantity.

FIG. 28 illustrates a configuration example of an image processingdevice according to a seventh embodiment.

FIG. 29 is a flowchart illustrating an image summarization processaccording to the seventh embodiment.

FIG. 30 is a view illustrating a summary candidate image sequencegeneration process.

FIG. 31 illustrates a configuration example of an image processingdevice according to an eighth embodiment.

FIGS. 32A to 32B are views illustrating the relationship between a scenechange and a partial image sequence.

FIG. 33 illustrates another configuration example of a deletiondetermination section.

FIG. 34 is a flowchart illustrating an image summarization processaccording to the eighth embodiment.

FIGS. 35A and 35B are views illustrating the relationship between theposition of a selected image within an image sequence and the positionof a detected scene change.

FIG. 36 illustrates a texture information calculation target areasetting example.

FIGS. 37A and 37B are views illustrating the difference in shape of acoverage area due to the difference in accuracy of deformationinformation.

FIG. 38 illustrates an example of an accuracy information calculationmethod.

FIG. 39 is a view illustrating the relationship between the positions ofa first image and a second image within an image sequence and theposition of a detected scene change.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

According to one embodiment of the invention, there is provided an imageprocessing device comprising: an image sequence acquisition section thatacquires an image sequence that includes a plurality of images; and aprocessing section that performs an image summarization process thatdeletes some of the plurality of images included in the image sequenceacquired by the image sequence acquisition section to acquire a summaryimage sequence, the processing section selecting a reference image and adetermination target image from the plurality of images, and determiningwhether or not the determination target image can be deleted based onresults of a process that utilizes deformation information about thereference image and the determination target image, and a process thatutilizes a structural element that corresponds to an attention area.

According to one embodiment of the invention, whether or not thedetermination target image can be deleted is determined based on theresults of the process that utilizes the deformation information, andthe process that utilizes the structural element (“the term “structuralelement” used herein is commonly also known in the art as a “structuringelement”). This makes it possible to link the structural element withthe attention area (i.e., an area that should not be missed), andimplement the image summarization process corresponding to the capturestate of the attention area within the reference image and thedetermination target image, for example.

According to another embodiment of the invention, there is provided animage processing device comprising: an image sequence acquisitionsection that acquires an image sequence that includes a plurality ofimages; and a processing section that performs an image summarizationprocess that deletes some of the plurality of images included in theimage sequence acquired by the image sequence acquisition section toacquire a summary image sequence, the processing section selecting areference image and a determination target image from the plurality ofimages, calculating a coverage area that is an area in which thedetermination target image is covered by the reference image based ondeformation information about the reference image and the determinationtarget image, and determining a probability that an attention area ismissed based on the coverage area to determine whether or not thedetermination target image can be deleted.

According to another embodiment of the invention, there is provided animage processing device comprising: an image sequence acquisitionsection that acquires an image sequence that includes a plurality ofimages; and a processing section that performs an image summarizationprocess that deletes some of the plurality of images included in theimage sequence acquired by the image sequence acquisition section toacquire a summary image sequence, the processing section detecting ascene change from the image sequence, setting a partial image sequencethat includes images among the plurality of images included in the imagesequence based on the detected scene change, selecting a reference imageand a determination target image from the partial image sequence, anddetermining whether or not the determination target image can be deletedbased on deformation information about the reference image and thedetermination target image.

According to this embodiment of the invention, the partial imagesequence is set based on a scene change detected from the imagesequence, and the deletion determination process that utilizes thedeformation information is performed on the partial image sequence. Thismakes it unnecessary to perform the process on the image included in agiven partial image sequence and the image that is not included in thegiven partial image sequence among the plurality of images included inthe image sequence, and implement an efficient image summarizationprocess, for example.

Another embodiment of the invention relates to a computer-readablestorage device with an executable program stored thereon, wherein theprogram instructs a computer to function as each of the above sections.

According to another embodiment of the invention, there is provided animage processing method comprising: acquiring an image sequence thatincludes a plurality of images; selecting a reference image and adetermination target image from the plurality of images included in theimage sequence; determining whether or not the determination targetimage can be deleted based on results of a process that utilizesdeformation information about the reference image and the determinationtarget image, and a process that utilizes a structural element thatcorresponds to an attention area; and performing an image summarizationprocess that deletes some of the plurality of images included in theimage sequence based on a result of the determination as to whether ornot the determination target image can be deleted to acquire a summaryimage sequence.

According to another embodiment of the invention, there is provided animage processing method comprising: acquiring an image sequence thatincludes a plurality of images; selecting a reference image and adetermination target image from the plurality of images included in theimage sequence; calculating a coverage area based on deformationinformation about the reference image and the determination targetimage, the coverage area being an area in which the determination targetimage is covered by the reference image; determining a probability thatan attention area is missed based on the coverage area to determinewhether or not the determination target image can be deleted; andperforming an image summarization process that deletes some of theplurality of images included in the image sequence based on a result ofthe determination as to whether or not the determination target imagecan be deleted to acquire a summary image sequence.

According to another embodiment of the invention, there is provided animage processing method comprising: acquiring an image sequence thatincludes a plurality of images; detecting a scene change from theacquired image sequence; setting a partial image sequence that includesimages among the plurality of images included in the image sequencebased on the detected scene change; selecting a reference image and adetermination target image from the partial image sequence; andperforming an image summarization process that determines whether or notthe determination target image can be deleted based on deformationinformation about the reference image and the determination targetimage, and deletes some of the plurality of images included in the imagesequence to acquire a summary image sequence.

Exemplary embodiments of the invention are described below. Note thatthe following exemplary embodiments do not in any way limit the scope ofthe invention laid out in the claims. Note also that all of the elementsdescribed in connection with the following exemplary embodiments shouldnot necessarily be taken as essential elements of the invention.

1. Method

A method employed in connection with several exemplary embodiments ofthe invention is described below. It is desirable to perform the imagesummarization process when an image sequence that includes a largenumber of temporally or spatially continuous images has been acquired,and the user performs a process (e.g., medical practice (e.g.,diagnosis) when the image sequence is an endoscopic image sequence)using the image sequence. This is because the number of images includedin the image sequence is very large, and it takes time for the user tocheck all of the images included in the image sequence to make adetermination. Moreover, it is likely that similar images are includedin the image sequence, and the amount of information that can beacquired is limited even if such similar images are thoroughly checked.

Specific examples of such an image sequence include an image sequencecaptured using a capsule endoscope. The capsule endoscope is acapsule-shaped endoscope that includes a small camera, and captures animage at given time intervals (e.g., twice a second). The capsuleendoscope remains inside a body for several hours (several tens of hoursin some cases) until the capsule endoscope is discharged from the body,and several tens of thousands of captured images are acquired during asingle examination. When the capsule endoscope moves inside a livingbody, the capsule endoscope may stop or move backward due to the motionof the living body, for example. Therefore, a large number of capturedimages may include a number of images that capture a similar object, andare not useful for finding a lesion or the like.

A known image summarization process may extract a scene change boundaryimage or an image that represents the image sequence. However, such aknown image summarization process does not take account of therelationship between the object captured within the deletion targetimage and the object captured within the image that is allowed to remainwhen deleting an image. Therefore, the object that is captured within animage included in the image sequence that is not subjected to the imagesummarization process may not be captured within each image included inthe image sequence obtained by the image summarization process.

This is particularly undesirable when applying the image summarizationprocess in the medical field. This is because it is necessary to preventa situation in which the attention area (e.g., lesion) is missed as muchas possible. In order to prevent a situation in which the attention areais missed, it is desirable to capture a wide range inside a living body,and prevent a situation in which an object range that cannot be observedoccurs due to deletion of a given image during the image summarizationprocess.

It is effective to use deformation information about images. Forexample, a reference image (i.e., an image that is allowed to remain (animage that may be allowed to remain depending on the reference imagesetting method)) and a determination target image (i.e., a deletiondetermination target image) may be selected from an image sequence, andthe image summarization process may be performed based on the coverageratio of the determination target image by the reference image.Specifically, the reference image is deformed to calculate a coveragearea within the determination target image (see FIG. 1). The objectcaptured within the reference image corresponds to the object capturedwithin the coverage area of the determination target image.Specifically, an area (hereinafter referred to as “non-coverage area”)of the determination target image other than the coverage area cannot becovered by the reference image when the determination target image isdeleted.

Therefore, the degree by which an object range that cannot be observedoccurs is controlled by calculating the ratio of the coverage area withrespect to the determination target image as the coverage ratio, anddetermining whether or not to delete the determination target imagebased on the calculated coverage ratio, for example. For example, thedetermination target image is deleted when the coverage ratio is equalto or larger than a threshold value, and is not deleted when thecoverage ratio is less than the threshold value. In this case, thedegree by which an that cannot be covered occurs can be controlledcorresponding to the threshold value.

However, the coverage ratio (e.g., the ratio of the coverage area withrespect to the determination target image) does not take account of theshape of the coverage area (or the non-coverage area). FIGS. 2A and 2Billustrate a determination target image in which the ratio of the areaof the coverage area to the area of the entire determination targetimage is 75% (i.e., the coverage ratio is 0.75 or 75). However, when theattention area has the size and the shape illustrated in FIG. 2C, thedetermination target image illustrated in FIG. 2A and the determinationtarget image illustrated in FIG. 2B differ to a large extent even if thecoverage ratio is the same. Specifically, the non-coverage areaillustrated in FIG. 2A cannot include the entire attention areaillustrated in FIG. 2C. Therefore, at least part of the attention areais necessarily captured within the reference image when the entireattention area is captured within the determination target imageillustrated in FIG. 2A irrespective of the position of the attentionarea. The non-coverage area illustrated in FIG. 2B can include theentire attention area illustrated in FIG. 2C. Therefore, the attentionarea is not captured within the reference image when the attention areais situated at the position indicated by the dotted line in FIG. 2B.

Specifically, when at least part of the attention area must be capturedwithin the reference image irrespective of the position of the attentionarea within the determination target image, the determination targetimage illustrated in FIG. 2A can be deleted, and the determinationtarget image illustrated in FIG. 2B cannot be deleted. However, thedetermination target image illustrated in FIG. 2A and the determinationtarget image illustrated in FIG. 2B cannot be distinguished by thedetermination process based on the coverage ratio since the coverageratio is the same.

In order to deal with the above problem, several aspects of theinvention propose a method that selects the reference image and thedetermination target image, and determines whether or not thedetermination target image can be deleted based on the results of aprocess that utilizes a structural element that corresponds to theattention area. Specifically, an erosion process that utilizes thestructural element is performed on the non-coverage area (describedlater with reference to FIGS. 6A to 6E) to determine whether or not aresidual area is present. When the residual area is not present, theentirety of an area having a size corresponding to that of the attentionarea cannot be included within the non-coverage area (see FIG. 2A, forexample), and it is determined that the determination target image canbe deleted. When the residual area is present, the entirety of an areahaving a size corresponding to that of the attention area may beincluded within the non-coverage area (see FIG. 2B, for example), and itis determined that the determination target image cannot be deleted.

This makes it possible to implement an image summarization process thatensures (or may ensure) that at least part of the attention area iscaptured within the reference image even when the determination targetimage is deleted. When using the above method, it is necessary to setthe shape and the size of the structural element. The structural elementmay have a shape and the like corresponding to those of the attentionarea. The shape and the like of the attention area differ correspondingto the imaging target and the situation. For example, when capturing animage using a capsule endoscope for a medical purpose, the minimum sizeof a lesion that should not be missed may be set to be the size of theattention area. In this case, at least part of a lesion having a sizeequal to or larger than the size (lesion size) of the attention area iscaptured within the reference image even when the determination targetimage is deleted, and it is possible to prevent a situation in which alesion having a large size (i.e., severe lesion) is missed.

The structural element may be information that indicates an image areaused for the erosion process. Note that the structural element is notlimited thereto. The term “structural element” used herein includesvarious types of information that is set based on the attention area,and used to determine whether or not the determination target image canbe deleted (see fourth embodiment). The structural element may have thesame shape and the same size as those of the attention area (see firstembodiment and fourth embodiment). In such a case, the structuralelement setting process is also performed to set an area having the sameshape and the same size as those of the attention area.

An image processing device according to one embodiment of the inventionmay include a processing section 100 and an image sequence acquisitionsection 200 (see FIG. 3). The image sequence acquisition section 200acquires an image sequence that includes a plurality of images. Theprocessing section 100 performs an image summarization process thatdeletes some of the plurality of images included in the image sequenceacquired by the image sequence acquisition section 200 to acquire asummary image sequence. Specifically, the processing section 100 selectsthe reference image and the determination target image from theplurality of images, and determines whether or not the determinationtarget image can be deleted based on the results of a process thatutilizes the deformation information about the reference image and thedetermination target image, and a process that utilizes the structuralelement that corresponds to the attention area.

A first embodiment illustrates a basic method. In the first embodiment,the process that selects the reference image and the determinationtarget image, and the process that determines whether or not thedetermination target image can be deleted based on the selectedreference image and the selected determination target image, areperformed. Note that these processes may be modified in various ways. Asecond embodiment illustrates another method that selects the referenceimage and the determination target image, and modifications thereof. Theprocess that determines whether or not the determination target imagecan be deleted may be implemented using a method that differs from themethod according to the first embodiment, and the details thereof aredescribed in connection with third to fifth embodiments.

The process that selects the reference image and the determinationtarget image may be implemented using the method according to the firstembodiment or the method according to the second embodiment (or themodification thereof), and the process that determines whether or notthe determination target image can be deleted may be implemented usingthe method according to the first embodiment, the method according tothe third embodiment, the method according to the fourth embodiment, orthe method according to the fifth embodiment. An arbitrary method thatselects the reference image and the determination target image, and anarbitrary method that determines whether or not the determination targetimage can be deleted may be appropriately combined. For example, themethod according to the second embodiment and the method according tothe third embodiment may be combined.

Even if the object area captured within the deletion target image iscaptured within the image that is allowed to remain, it may be difficultto take account of ease of observation of the object area within theimage that is allowed to remain. In FIG. 26, the object that is capturedwithin a given area of the determination target image is captured withina very narrow area of the reference image. In the example illustrated inFIG. 26, it may be determined based on the coverage ratio that theobject is captured within the reference image, and is not missed even ifthe determination target image is deleted (i.e., it may be determinedthat the determination target image can be deleted), for example.However, since the size of the object within the reference image is verysmall (see FIG. 26), the reference image illustrated in FIG. 26 may notbe suitable for observation depending on the number of pixels(resolution) of the reference image. If the size of the object is small,the object may be missed. Even if the object is not missed, it isdifficult to sufficiently observe the object when it is necessary toclosely observe the object in order to determine whether or not theobject is a lesion, or determine the degree of progression of thelesion, for example.

In order to deal with the above problem, several aspects of theinvention propose a method that selects the reference image and thedetermination target image, and determines whether or not thedetermination target image can be deleted based on the feature quantityof an observation area within the determination target image, and thefeature quantity of a corresponding area that is an area within thereference image that corresponds to the observation area. In this case,the corresponding area is calculated by deforming the observation areabased on the deformation information about the reference image and thedetermination target image. Specifically, the object area capturedwithin the observation area corresponds to (coincides with in a narrowsense) the object area captured within the corresponding area.

For example, a situation in which the size of the corresponding area issmaller than that of the observation area (see FIG. 26) can be detectedby utilizing size information (e.g., area) about the area as the featurequantity. This makes it possible to deal with the above problem.Specifically, since the object is captured within a narrow range (i.e.,a range corresponding to the corresponding area) within the referenceimage, and is not suitable for observation, it is determined that thedetermination target image cannot be deleted (i.e., the determinationtarget image is allowed to remain in the summary image sequence), andthe object is observed in a state in which the object is captured over awide range (i.e., a range corresponding to the observation area) withinthe determination target image.

A sixth embodiment illustrates a basic method that implements the imagesummarization process that determines whether or not the determinationcan be deleted based on the feature quantity of the observation area andthe feature quantity of the corresponding area. Note that the imagesummarization process may be performed while combining the process thatutilizes the observation area with the process that utilizes thestructural element or the like. A seventh embodiment illustrates aspecific combination method.

A modification may be made that takes account of an improvement in speedand accuracy of the process that utilizes the deformation information.For example, it may be inefficient to select the reference image and thedetermination target image from the entire processing target imagesequence. Specifically, when the imaging target differs to a largeextent between the first part and the second part of the processingtarget image sequence (e.g., when the first part and the second part ofan image sequence captured by a capsule endoscope respectively capturethe stomach and the small intestine), it is not considered that theimages in the second part are covered by the images in the first part.Therefore, it may be unnecessary to perform a comparison process on thefirst part and the second part. In this case, an improvement inefficiency can be achieved by separately performing the imagesummarization process on the first part and the image summarizationprocess on the second part.

Several aspects of the invention thus propose a method that detects ascene change from the image sequence, and divides the image sequenceinto a plurality of partial image sequences based on the detected scenechange. The image summarization process that utilizes the deformationinformation may be independently performed on each partial imagesequence. This makes it possible to efficiently perform the imagesummarization process. Moreover, since the image summarization processcan be performed on a plurality of partial image sequences in parallel,the processing speed can be increased. The details thereof are describedin connection with an eighth embodiment.

2. First Embodiment

The basic method according to the first embodiment is described below. Asystem configuration example of the image processing device will bedescribed first, and the flow of the process will then be describedusing a flowchart.

FIG. 4 illustrates a system configuration example of the imageprocessing device according to the first embodiment. The imageprocessing device includes a processing section 100, an image sequenceacquisition section 200, and a storage section 300.

The processing section 100 performs an image summarization process on animage sequence acquired by the image sequence acquisition section 200,the image summarization process deleting some of a plurality of imagesincluded in the image sequence. The function of the processing section100 may be implemented by hardware such as a processor (e.g., CPU) or anASIC (e.g., gate array), a program, or the like.

The image sequence acquisition section 200 acquires the image sequencethat is subjected to the image summarization process. The storagesection 300 stores the image sequence acquired by the image sequenceacquisition section 200, and serves as a work area for the processingsection 100 and the like. The function of the storage section 300 may beimplemented by a memory (e.g., RAM), a hard disk drive (HDD), or thelike.

The processing section 100 may include a reference image selectionsection 1001, a determination target image selection section 1002, acoverage area calculation section 1003, a deletion determination section1005, a partial image sequence setting section 1008, a summary imagesequence determination section 1009, an attention area miss probabilitydetermination section 1013, a structural element generation section1014, and a deformation information acquisition section 1015 (see FIG.4). Note that the configuration of the processing section 100 is notlimited to the configuration illustrated in FIG. 4. Variousmodifications may be made, such as omitting some of the elementsillustrated in FIG. 4, or adding other elements. Note that the abovesections are set to describe each subroutine when the imagesummarization process performed by the processing section 100 is dividedinto a plurality of subroutines. The processing section 100 does notnecessarily include the above sections as elements.

The reference image selection section 1001 selects a reference imagefrom the plurality of images included in the image sequence. Thedetermination target image selection section 1002 selects an image amongthe plurality of images included in the image sequence that differs fromthe reference image as a determination target image.

The coverage area calculation section 1003 projects the reference imageonto the determination target image by utilizing deformation information(deformation parameter) about the reference image and the determinationtarget image to calculate the coverage area. An area of thedetermination target image other than the coverage area may be set to bea non-coverage area.

The deletion determination section 1005 determines whether or not thedetermination target image can be deleted based on the determinationresult of the attention area miss probability determination section 1013(described later).

The partial image sequence setting section 1008 sets an image sequencethat is included in the image sequence and includes one or more imagesto be a partial image sequence based on the position of thedetermination target image in the image sequence when the deletiondetermination section 1005 has determined that the determination targetimage cannot be deleted. Note that the term “partial image sequence”used in connection with the first to seventh embodiments refers to theunit of repetition processing, and differs from the partial imagesequence used in connection with the eighth embodiment that is thetarget of parallel processing.

The summary image sequence determination section 1009 determines asummary image sequence that is an image sequence obtained by thesummarization process. In the first embodiment, the reference imageselected by the reference image selection section 1001 is included inthe summary image sequence. The determination target image that can bedeleted is deleted, and is not included in the summary image sequence.

The attention area miss probability determination section 1013 performsa determination process that determines the probability that theattention area captured within the determination target image is notcaptured within the reference image (i.e., the attention area is missed)when the determination target image is deleted. The details of thedetermination process are described later.

The structural element generation section 1014 generates a structuralelement used for the process performed by the attention area missprobability determination section 1013 based on the attention area. Forexample, an area having the same shape and the same size as those of theattention area is set to be the structural element. Note that thestructural element is not limited thereto.

The deformation information acquisition section 1015 acquires thedeformation information about two images. The deformation informationindicates a shape (range) in which the range captured within one imageis captured within the other image. The deformation information may bethe deformation parameter disclosed in JP-A-2011-24763, for example. Thedeformation information acquisition section 1015 acquires thedeformation information about the reference image selected by thereference image selection section 1001 and the determination targetimage selected by the determination target image selection section 1002.

FIG. 5 is a flowchart illustrating the image summarization processaccording to the first embodiment. When the image summarization processhas started, the image sequence that is subjected to the imagesummarization process is acquired (S101). The image sequence is acquiredby the image sequence acquisition section 200. The image sequence mayinclude RGB channel images that are arranged in time series.Alternatively, the image sequence may be a spatially consecutive imagesequence (e.g., an image sequence that includes spatially arrangedimages that have been captured using imaging devices arranged in a row).Note that the images included in the image sequence are not limited toRGB channel images. Another color space (e.g., gray channel image) mayalso be used.

The structural element used for the image summarization process (i.e.,determination as to whether or not the determination target image can bedeleted) is generated based on the attention area (S102). In the firstembodiment, an area having the same shape and the same size as those ofthe attention area is set to be the structural element. Note thatanother method may also be used. For example, a quadrilateral(rectangular) area that is circumscribed to a circular attention areamay be set to be the structural element. In particular, when the shapeof the attention area is complex, the amount of calculations can bereduced by setting the structural element using a simple shape, forexample.

The reference image selection section 1001 selects the first image ofthe input image sequence (i.e., the image sequence acquired in the stepS101, or the partial image sequence set in a step S109) as the referenceimage (S103). The selected reference image is allowed to remain in thesummary image sequence. Note that the process is terminated when thereference image cannot be selected from the input image sequence (e.g.,when no image is included in the image sequence) due to an error or thelike.

The determination target image selection section 1002 selects thedetermination target image from the images included in the input imagesequence (S104). When the determination target image has not been set,the image that immediately follows the reference image (i.e., the secondimage of the input image sequence) is selected as the determinationtarget image. When the kth image of the input image sequence has beenselected as the determination target image, the (k+1)th image (i.e., theselection position is shifted by 1) of the input image sequence isselected as the next determination target image. The process isterminated when the determination target image cannot be selected (e.g.,when the number of images included in the input image sequence is lessthan 2 or k+1).

When the reference image and the determination target image have beenselected, the deformation information acquisition section 1015 acquiresthe deformation information about the reference image and thedetermination target image (S105). The coverage area calculation section1003 projects the reference image onto the determination target image byutilizing the acquired deformation information to calculate the coveragearea (S106). The deformation information may be a non-rigid deformationparameter estimated by the method disclosed in JP-A-2011-24763, forexample. FIG. 1 illustrates an example of the coverage area. Thedeformation parameter represents a state in which the object capturedwithin the reference image is deformed within the determination targetimage. In other words, the object captured within the reference imagecorresponds to (identical with in a narrow sense) the object capturedwithin the coverage area of the determination target image.

When the coverage area has been calculated, the attention area missprobability determination section 1013 determines the probability thatthe attention area is missed (S107). Specifically, the attention areamiss probability determination section 1013 performs an erosion processthat utilizes the structural element on the non-coverage area of thedetermination target image other than the coverage area to determinewhether or not a residual area is present.

A specific example of the erosion process is described below withreference to FIGS. 6A to 6E. As illustrated in FIG. 6A, the non-coveragearea is necessarily a closed area, and the boundary of the non-coveragearea can be set. For example, an outer boundary BO1 and an innerboundary BO2 are set in FIG. 6A.

The erosion process that utilizes the structural element removes theoverlapping area of the non-coverage area and the structural elementwhen a reference point of the structural element is set at the boundaryof the non-coverage area. For example, when a circular area (see FIG.2C) is set to be the structural element, and the reference point of thestructural element is the center of the circle, the erosion processdraws a circle so that the center of the circle is situated at theboundary of the non-coverage area, and excludes the overlapping area ofthe circle and the non-coverage area from the non-coverage area.Specifically, a circle is drawn around a point situated at the outerboundary BO1 of the non-coverage area (see FIG. 6A), and the overlappingarea of the circle and the non-coverage area (i.e., the semicirculararea indicated by the diagonal lines in FIG. 6A) is excluded from thenon-coverage area.

Since the outer boundary BO1 is processed discretely, and includes aplurality of points, the above process may be performed on each pointamong the plurality of points. For example, a circle may be sequentiallydrawn around each point situated at the outer boundary BO1 in a givendirection (see FIG. 6A), and the overlapping area of each circle and thenon-coverage area may be excluded from the non-coverage area.

The non-coverage area may have only a single boundary depending on theshape of the non-coverage area (see FIG. 2B). In such a case, the aboveprocess may be performed on the single boundary. When the non-coveragearea has the outer boundary BO1 and the inner boundary BO2 (see FIG.6A), the above process is performed on the outer boundary BO1 and theinner boundary BO2. Specifically, a circle is drawn around each pointsituated at the inner boundary BO2 (see FIG. 6B), and the overlappingarea of each circle and the non-coverage area is excluded from thenon-coverage area.

The non-coverage area decreases in area through the erosion process. Forexample, the left part of the non-coverage area illustrated in FIG. 6Ais completely deleted (i.e., no residual area is present) by the erosionprocess performed on the outer boundary BO1 (see FIG. 6A) and theerosion process performed on the inner boundary BO2 (see FIG. 6B). Onthe other hand, a residual area RE that is not excluded by the erosionprocess performed on the outer boundary BO1 and the erosion processperformed on the inner boundary BO2 occurs in the lower right part ofthe non-coverage area (see FIG. 6C). Specifically, only the residualarea RE remains as a result of performing the erosion process thatutilizes the structural element over the entire non-coverage area (seeFIG. 6D).

The meaning of the erosion process when using a circle having a radius ras the structural element is discussed below. The non-coverage area(i.e., closed area) is considered to be an area that is surrounded by aboundary (different boundaries (e.g., BO1 and BO2) or a single boundary(see FIG. 2B)). When the erosion process is performed on the boundary, apoint among the points included in the non-coverage area that issituated at a distance equal to or shorter than r from each pointsituated at the boundary is determined to be the deletion target.Specifically, the distance from the point included in the residual area(that is excluded from the deletion target) to an arbitrary pointsituated at the boundary is longer than r. Therefore, a circle having aradius r that is drawn around an arbitrary point within the residualarea does not intersect each boundary. This means that the entirety ofthe attention area represented by a circle having a radius R (=r) thatis drawn around a point within the residual area is included within thenon-coverage area. Note that the above basic idea is also applied evenwhen the structural element has a shape (e.g., quadrangle) other than acircle.

Specifically, when the residual area is present, an area thatcorresponds to the structural element is included within thenon-coverage area (see the lower right part in FIG. 6E). When theattention area (e.g., lesion) is situated at such a position, and thedetermination target image is deleted, it is likely that the attentionarea cannot be observed even if the reference image is allowed toremain. When the residual area is not present, at least part of theattention area is included within the coverage area (see the upper leftpart in FIG. 6E). In this case, at least part of the attention arearemains in the reference image even if the determination target image isdeleted. Therefore, whether or not the determination target image can bedeleted can be determined by causing the attention area miss probabilitydetermination section 1013 to perform the erosion process that utilizesthe structural element on the non-coverage area, and output the presenceor absence of the residual area as the determination result.

The deletion determination section 1005 determines whether or not thedetermination target image can be deleted based on whether or not theresidual area is present as a result of the erosion process (S108). Whenthe residual area is present, the entire attention area may be includedwithin the non-coverage area (i.e., the attention area may be missed).In this case, it is determined that the determination target imagecannot be deleted, and a partial image sequence setting process isperformed. When the residual area is not present, the entire attentionarea is not included within the non-coverage area, and at least part ofthe attention area is included within the coverage area. In this case,it is not likely that the attention area is missed. Therefore, it isdetermined that the determination target image can be deleted, andanother image is selected as the determination target image in the stepS104.

When it has been determined that the determination target image cannotbe deleted in the step S108, the partial image sequence setting section1008 sets the partial image sequence (S109). Specifically, an imagesequence that includes the determination target image that cannot bedeleted, and the subsequent images is set to be the partial imagesequence. When the partial image sequence has been set, the process inthe step S103 is performed using the partial image sequence as the inputimage sequence.

FIGS. 7A to 7D illustrate the image summarization process. When an imagesequence that includes N images (see FIG. 7A) has been acquired by theimage sequence acquisition section 200, the first image is selected asthe reference image, and the second image is selected as thedetermination target image. The probability that the attention area ismissed is determined using the reference image and the determinationtarget image, and whether or not the determination target image can bedeleted is determined.

When it has been determined that the determination target image can bedeleted, another image is selected as the determination target image.Specifically, the third image is selected as the determination targetimage (i.e., the position of the determination target image is shiftedto the subsequent image) (see FIG. 7B). Whether or not the determinationtarget image can be deleted is determined using the reference image andthe new determination target image, and another image selected as thedetermination target image until it is determined that the determinationtarget image can be deleted.

When it has been determined that the second to (k−1)th images can bedeleted (i.e., the second to (k−1)th images are covered by the referenceimage to such an extent that the attention area is not missed), and thekth image cannot be deleted (see FIG. 7C), the second to (k−1)th imagesare deleted (i.e., the second to (k−1)th images are not included in thesummary image sequence). Since the kth image is not sufficiently coveredby the reference image, it is necessary to allow the kth image to remainin the summary image sequence. Therefore, the kth image and thesubsequent images (kth to Nth images) are set to be the partial imagesequence.

The process illustrated in FIGS. 7A to 7C is then performed on thepartial image sequence. Specifically, the partial image sequence thatincludes N−x+1 images is used as the input image sequence (see FIG. 7D),and the process is performed using the first image (i.e., the kth imagein FIG. 7C) as the reference image, and using the second image (i.e.,the (k+1)th image in FIG. 7C) as the determination target image. Thesubsequent process is performed in the same manner as described above.When it has been determined that the determination target image can bedeleted, the subsequent image is selected as the determination targetimage. When it has been determined that the determination target imagecannot be deleted, the reference image is allowed to remain in thesummary image sequence, the image that can be deleted is deleted, andthe current determination target image and the subsequent images are setto be a new partial image sequence. The process is terminated when ithas been determined that the last image of the input image sequence canbe deleted, or when only one image is included in the input imagesequence (i.e., when the determination target image cannot be set).

According to the first embodiment, the image processing device includesthe image sequence acquisition section 200 that acquires an imagesequence that includes a plurality of images, and the processing section100 that performs the image summarization process that deletes some ofthe plurality of images included in the image sequence acquired by theimage sequence acquisition section 200 to acquire a summary imagesequence (see FIG. 4). The processing section 100 selects the referenceimage and the determination target image from the plurality of images,and determines whether or not the determination target image can bedeleted based on the results of a process that utilizes the deformationinformation about the reference image and the determination targetimage, and a process that utilizes the structural element thatcorresponds to the attention area.

The process that utilizes the deformation information may be a processthat deforms at least one of the reference image and the determinationtarget image using the deformation information. The process thatutilizes the structural element that corresponds to the attention areamay be an erosion process that utilizes the structural element, or aprocess that determines whether or not the structural element isincluded within the non-coverage area, the non-coverage area being anarea in which the determination target image is not covered by thereference image.

The term “attention area” used herein refers to an area for which theuser's observation priority is relatively higher than that of otherareas. For example, when the user is a doctor, and desires to performtreatment, the attention area refers to an area that includes a mucousmembrane area or a lesion area. If the doctor desires to observe bubblesor feces, the attention area refers to an area that includes a bubblearea or a feces area. Specifically, the attention area for the userdiffers depending on the objective of observation, but is necessarily anarea for which the user's observation priority is relatively higher thanthat of other areas.

According to this configuration, the process that utilizes thedeformation information can be performed, and it is possible toimplement the image summarization process that takes account of therelationship between the object captured within the reference image andthe object captured within the determination target image. It ispossible to implement the process corresponding to the capture state ofthe attention area by performing the process that utilizes thestructural element that corresponds to the attention area. Specifically,since whether or not at least part of the attention area captured withinthe determination target image is captured within the reference imagecan be determined by performing the erosion process that utilizes thestructural element or the like, it is possible to allow an image inwhich at least part of the attention area is captured to necessarilyremain in the summary image sequence even if the determination targetimage is deleted. This makes it possible to reduce the possibility thatthe user misses the attention area, for example.

The processing section 100 may perform a process that deforms thereference image using the deformation information to calculate thecoverage area, and calculates an area of the determination target imageother than the coverage area as the non-coverage area as the processthat utilizes the deformation information, the coverage area being anarea in which the determination target image is covered by the referenceimage. The processing section 100 may perform the erosion process thatutilizes the structural element on the non-coverage area as the processthat utilizes the structural element. The processing section 100 maydetermine that the determination target image cannot be deleted when aresidual area is present as a result of the erosion process.

This makes it possible to accurately determine whether or not theentirety of the structural element (having the same size as that of theattention area) is included within the non-coverage area. Since thenon-coverage area is an area of the determination target image that isnot covered by the reference image, the presence or absence of theresidual area corresponds to whether or not the entirety of thestructural element is included within the non-coverage area (see FIGS.6A to 6E). Since whether or not the entirety of the structural elementis included within the non-coverage area is strictly determined, theaccuracy of the determination as to whether or not the determinationtarget image can be deleted can be increased as compared with the methodaccording to the third embodiment and the method according to the fourthembodiment (described later), for example.

The processing section 100 may calculate the deformation informationabout adjacent images among the reference image, the determinationtarget image, and an image among a plurality of images that is situatedbetween the reference image and the determination target image, andcalculate the deformation information about the reference image and thedetermination target image based on the calculated deformationinformation about the adjacent images.

According to this configuration, when the reference image and thedetermination target image are not adjacent to each other, it ispossible to calculate the deformation information about the referenceimage and the determination target image by accumulating the deformationinformation about adjacent images instead of directly calculating thedeformation information about the reference image and the determinationtarget image. The deformation information can be calculated using themethod disclosed in JP-A-2011-24763, for example. The processing loadimposed by a process that combines a plurality of pieces of deformationinformation is normally very low as compared with a process thatcalculates the deformation information from the beginning. For example,when the deformation information is a matrix, the processing loadimposed by a process that calculates the matrix from two pieces of imageinformation is heavy, while it is very easy to synthesize a plurality ofmatrices calculated in advance (since it suffices to calculate theproduct of the matrices).

This method is particularly effective when implementing a process thatutilizes the deformation information a number of times (see the secondembodiment), for example. In the second embodiment, the reference image(second reference image) that follows the determination target image isalso set in addition to the reference image that precedes thedetermination target image, and the second reference image is updatedcorresponding to the conditions. Specifically, when the first image isused as the first reference image, and the kth image is used as thesecond reference image, whether or not the determination target imagecan be deleted is determined using the second to (k−1)th images and eachreference image, and the second reference image is updated with the(k+1)th image depending on the conditions without updating the firstreference image. In this case, the deformation information about each ofthe second to (k−1)th images and the (k+1)th image (second referenceimage) is required, and it is necessary to calculate the deformationinformation k−1 times. Since the required deformation informationdiffers from the deformation information about each of the second to(k−1)th images and the kth image (preceding second reference image), itis necessary to additionally calculate the required deformationinformation. For example, when the image sequence acquisition section200 acquires N images as the image sequence, and the second referenceimage is sequentially updated with the third to Nth images while thefirst reference image remains unchanged, it is necessary to calculatethe deformation information 1+2+3+ . . . +(N−2)=(N−2)(N−1)/2 times.Specifically, it is necessary to perform the deformation informationcalculation process that imposes a heavy load a number of times, and itis inefficient.

When using the deformation information about adjacent images, itsuffices to calculate the deformation information N−1 times when theimage sequence acquisition section 200 has acquired N images as theimage sequence. In this case, it is necessary to perform a process thatsynthesizes pieces of deformation information among the N−1 pieces ofdeformation information when the reference image and the determinationtarget image have been selected from the N images. However, theprocessing load imposed by the synthesis process is low as compared withthe deformation information calculation process.

The processing section 100 may set the structural element having a sizeproportional to the size of the attention area.

This makes it possible to link the size of the attention area with thesize of the structural element. In the first embodiment, the size of theattention area and the size of the structural element may be almostidentical (identical in a narrow sense). In the third embodiment(described later), the size of the structural element is set to be twicethe size of the attention area, for example. Since whether or not thedetermination target image can be deleted is determined based on whetheror not the entirety of an area having the same size as that of theattention area is included within the non-coverage area (or an area thatcorresponds to the non-coverage area), the ratio of the size of theattention area to the size of the structural element is determinedcorresponding to the process.

When first to Nth (N is an integer equal to or larger than 2) imageshave been input as the input image sequence, the processing section 100may select the first image as the reference image, and select the kth (kis an integer that satisfies 2≤k≤N−1) image as the determination targetimage. The processing section 100 may determine whether or not thedetermination target image can be deleted based on the deformationinformation about the reference image and the determination target imageand the process that utilizes the structural element that corresponds tothe attention area. The processing section 100 may select the (k+1)thimage as the determination target image when it has been determined thatthe kth image can be deleted.

The input image sequence refers to an image sequence that is subjectedto the above process (selection of the reference image and thedetermination target image, determination as to whether or not thedetermination target image can be deleted, and update of thedetermination target image when the determination target image can bedeleted), and may be an image sequence acquired by the image sequenceacquisition section 200, or may be an image sequence that includesimages among the images included in the image sequence acquired by theimage sequence acquisition section 200.

This makes it possible to implement the process illustrated in FIGS. 7Aand 7B and the like when the input image sequence has been input. Thedetermination target image is selected from the images that follow thereference image (e.g., the image that is adjacent to the referenceimage), and the determination target image is updated with thesubsequent image when the selected determination target image can bedeleted. Specifically, whether or not the determination target image canbe deleted is determined sequentially from the image situated closer tothe reference image to determine (search) an image that cannot bedeleted. The determination target image that can be deleted is basicallydeleted, and is not included in the summary image sequence. Note thatthe configuration is not limited thereto. A determination target imageamong the determination target images that can be deleted may beincluded in the summary image sequence.

The processing section 100 may perform a process that allows the imageselected as the reference image to be included in the summary imagesequence. The processing section 100 may set the partial image sequencethat includes the kth to Nth images to be the next input image sequencewhen it has been determined that the kth image selected as thedetermination target image cannot be deleted, and may perform theprocess on the set input image sequence.

This makes it possible to implement the process illustrated in FIG. 7C.Since the process that determines whether or not the determinationtarget image can be deleted is performed based on the erosion processthat utilizes the structural element, at least part of the attentionarea can be observed by allowing the reference image to remain, even ifthe determination target image that can be deleted, is deleted.Specifically, the reference image is included in the summary imagesequence. When it has been determined that the kth image cannot bedeleted (i.e., when the kth image cannot be sufficiently covered by thereference image), it is necessary to allow the kth image to remain inthe summary image sequence. Specifically, the kth image may be set to bethe next reference image. For example, the partial image sequence thatincludes the kth to Nth to images may be set to be the next input imagesequence. According to this configuration, the first image of the inputimage sequence (i.e., the kth image included in the image sequenceacquired by the image sequence acquisition section 200) is selected asthe reference image, and the (k+1)th and subsequent images aresequentially selected as the determination target image (see FIG. 7D).Note that the partial image sequence that includes images among theimages included in the image sequence illustrated in FIG. 7D may be setto be the input image sequence. The process is repeated (or recursively)performed.

The image processing device according to the first embodiment mayinclude the image sequence acquisition section 200 that acquires animage sequence that includes a plurality of images, and the processingsection 100 that performs the image summarization process that deletessome of the plurality of images included in the image sequence acquiredby the image sequence acquisition section 200 to acquire a summary imagesequence, the processing section 100 may select the reference image andthe determination target image from the plurality of images, calculatethe coverage area that is an area in which the determination targetimage is covered by the reference image based on the deformationinformation about the reference image and the determination targetimage, determine the probability that the attention area is missed basedon the coverage area, and determine whether or not the determinationtarget image can be deleted based on the determination result for theprobability that the attention area is missed.

The probability that the attention area is missed refers to theprobability that the attention area is not observed by the user. Forexample, when the attention area captured within a given image includedin the image sequence that is not subjected to the image summarizationprocess is not captured within each image included in the summary imagesequence obtained by the image summarization process, it is impossiblefor the user who observes the summary image sequence to observe theattention area. In this case, the probability that the attention area ismissed is 100%. When the attention area is captured within the imageincluded in the summary image sequence, the probability that theattention area is missed may be determined to be 0%. Specifically, theprobability that the attention area is missed used herein is simplybased on whether or not the attention area is captured within the imageincluded in the summary image sequence, and does not take account of asituation in which the user erroneously misses the attention area evenwhen the attention area is captured within the image included in thesummary image sequence. Note that the probability that the attentionarea is missed may be a value between 0% and 100%. For example, theratio of the area of the attention area within the image included in thesummary image sequence to the area of the entire attention area may beused as the probability that the attention area is missed.

This makes it possible to implement the image summarization process thattakes account of the probability that the attention area is missed.Since the attention area is a lesion or the like in the medical field,it is very useful to suppress a situation in which the attention area ismissed.

Note that part or most of the processes performed by the imageprocessing device and the like according to the first embodiment may beimplemented by a program. In this case, the image processing device andthe like according to the first embodiment are implemented by causing aprocessor (e.g., CPU) to execute a program. Specifically, a programstored in an information storage medium is read, and executed by aprocessor (e.g., CPU). The information storage medium (computer-readablemedium) stores a program, data, and the like. The function of theinformation storage medium may be implemented by an optical disk (e.g.,DVD or CD), a hard disk drive (HDD), a memory (e.g., memory card orROM), or the like. The processor (e.g., CPU) performs various processesaccording to the first embodiment based on a program (data) stored inthe information storage medium. Specifically, a program that causes acomputer (i.e., a device including an operation section, a processingsection, a storage section, and an output section) to function as eachsection according to the first embodiment (i.e., a program that causes acomputer to execute the process implemented by each section) is storedin the information storage medium.

3. Second Embodiment

Another method that selects the reference image and the determinationtarget image is described below. In the second embodiment, a firstreference image and a second reference image are selected as thereference image. A basic method and two modifications are describedbelow. Note that the second embodiment differs from other embodiments asto the method that selects the reference image and the determinationtarget image, and the deletion determination process based on theselected reference image and the selected determination target image isimplemented using the method described in connection with anotherembodiment. Therefore, detailed description of the deletiondetermination process are omitted.

3.1 Basic Method According to Second Embodiment

FIG. 8 illustrates a system configuration example of an image processingdevice according to the second embodiment. In FIG. 8, the processingsection 100 further includes a second reference image selection section1007 in addition to those illustrated in FIG. 4.

The reference image selection section 1001 selects the first referenceimage. The second reference image selection section 1007 selects animage that follows the first reference image at an interval of one ormore images as the second reference image. The determination targetimage selection section 1002 selects an image that follows the referenceimage and precedes the second reference image as the determinationtarget image. Although FIG. 8 illustrates an example in which thereference image selection section 1001 and the second reference imageselection section 1007 are separately provided, the configuration is notlimited thereto. The reference image selection sections 1001 may selectboth the first reference image and the second reference image.

FIG. 9 is a flowchart illustrating the image summarization processaccording to the second embodiment. Note that steps S201 to S203 are thesame as the steps S101 to S103, respectively. After the step S203, animage that follows the first reference image selected in the step S203at an interval of one or more images is selected as the second referenceimage (S210). The determination target image is set (S204). When thedetermination target image has not been set, the image that immediatelyfollows the reference image (i.e., the second image of the input imagesequence) is selected as the determination target image. When the kthimage of the input image sequence has been selected as the determinationtarget image, the (k+1)th image (i.e., the selection position is shiftedby 1) of the input image sequence is selected as the next determinationtarget image. Note that the determination target image selection rangeis limited to the second reference image instead of the last image ofthe input image sequence.

When the determination target image does not coincide with the secondreference image, the deformation information is acquired (S205), and thecoverage area is calculated based on the acquired deformationinformation (S206). Specifically, a first candidate area is calculatedbased on the deformation parameter about the first reference image andthe determination target image, and a second candidate area iscalculated based on the deformation parameter about the second referenceimage and the determination target image. An area corresponding to thesum set of the first candidate area and the second candidate area is setto be the coverage area (see FIG. 10). Since the method according to thesecond embodiment determines whether or not the determination targetimage is sufficiently covered when both the first reference image andthe second reference image are allowed to remain, an area that iscovered by at least one of the first candidate area and the secondcandidate area can be set to be the coverage area. Specifically, an areathat is not included in the first candidate area and the secondcandidate area (i.e., an area of the determination target image otherthan the coverage area (see the first embodiment)) is set to be thenon-coverage area.

Note that steps S207 to S209 are the same as the steps S107 to S109,respectively. When it has been determined that the determination targetimage can be deleted in the step S208, the determination target image isupdated with the image that immediately follows the currentdetermination target image (S204). When the updated determination targetimage coincides with the second reference image, the second referenceimage is updated with the image that immediately follows the currentsecond reference image (S210). When the second reference image has beenupdated, the selected state of the determination target image is reset.When the determination target image does not coincide with the secondreference image, the process in the step S205 is performed.

When it has been determined that the determination target image cannotbe deleted in the step S208 (i.e., all of the images situated betweenthe first reference image and the current second reference image cannotbe covered by the first reference image and the current second referenceimage), the image that immediately precedes the current second referenceimage must be allowed to remain in the summary image sequence.Therefore, an image sequence that includes the image that immediatelyprecedes the current second reference image, and the subsequent imagesare set to be the partial image sequence (S209), and the process in thestep S203 is performed.

FIGS. 11A and 11B illustrate the image summarization process. In FIGS.11A and 11B, the kth image of the image sequence is selected as thefirst reference image. The first to (k−1)th images have been processed,and the kth to Nth mages have been set to be the partial image sequence.The (k+2)th image is selected as the second reference image.

The image situated between the first reference image and the secondreference image is selected as the determination target image, andwhether or not the determination target image can be deleted isdetermined.

When it has been determined that all of the images situated between thefirst reference image and the second reference image can be deleted (seeFIG. 11A) (i.e., the image that follows the second reference image canbe selected as the second reference image), the second reference imageis selected again as illustrated in FIG. 11B. Specifically, the (k+3)thimage is selected as the second reference image.

Whether or not the images situated between the first reference image andthe second reference image can be deleted is then determined. When ithas been determined that the determination target image cannot bedeleted (see FIG. 11B) (i.e., all of the images situated between thefirst reference image and the current second reference image cannot becovered by the first reference image and the current second referenceimage) (i.e., the determination target image that cannot be deletedcannot be covered by the first reference image and the current secondreference image), it is considered that the update of the secondreference image (increment in the selection position) was inappropriate.

Therefore, an image sequence that includes the image that immediatelyprecedes the current second reference image (corresponding to the secondreference image in FIG. 11A), and the subsequent images are set to bethe partial image sequence. Therefore, the second reference image inFIG. 11A is selected as the reference image during the subsequentprocess to ensure that the image to be deleted is covered by the imagethat is allowed to remain in the summary image sequence.

Although an example in which the first image of the input image sequenceis selected as the reference image in the step S203, the first image ofthe input image sequence need not necessarily be selected as thereference image when the process in the step S203 is performed for thefirst time. In the second embodiment, the determination target image canbe deleted as long as the determination target image is covered by thesecond reference image that follows the determination target image. Forexample, when the first image and the second image are covered by thethird image (see FIG. 12A), the first image and the second image neednot be allowed to remain in the summary image sequence. Therefore, thefirst image need not necessarily be allowed to remain in the summaryimage sequence. When the first image is used as the reference image, thenumber of images included in the summary image sequence mayunnecessarily increase.

In the second embodiment, the first reference image need not necessarilybe the first image of the image sequence acquired by the image sequenceacquisition section 200. An example of a specific method is describedbelow. As illustrated in FIG. 12B, a virtual zeroth image is actuallyselected as the first reference image. Note that this selection processis performed for convenience, and it is unnecessary to actually providethe zeroth image, for example. In this case, the second image isselected by the second reference image selection process in the stepS210, and the image (first image) situated between the first referenceimage and the second reference image is sequentially selected as thedetermination target image. The process in the steps S206 to 208 isperformed on the determination target image and the second referenceimage since the first reference image is not actually present. When thefirst image is covered by the second image, the second reference imageis updated with the third image according to the process illustrated inFIG. 9 (see FIG. 12C), and whether or not the first image and the secondimage are covered by the third image is determined. The kth image thatcannot cover all of the first to (k−1)th images when the kth image isselected as the second reference image (i.e., the (k−1)th image cancover all of the first to (k−2)th images when the (k−1)th image isselected as the second reference image) (see FIG. 12D) can be found byrepeating the above process. In this case, it is determined that thedetermination target image cannot be deleted in the step S208, an imagesequence that includes the (k−1)th to Nth images is set to be thepartial image sequence, and the process in the step S203 is performedagain. In this case, since the first image of the input image sequenceis selected as the reference image during the process in the step S203,the (k−1)th image is allowed to remain in the summary image sequence asthe reference image. Since the first to (k−2)th images are covered bythe (k−1)th image, the first to (k−2)th images can be deleted, and thenumber of images included in the summary image sequence can be reduced.

According to the second embodiment, when the first to Nth (N is aninteger equal to or larger than 2) images have been input as the inputimage sequence, the processing section 100 selects the pth image as thefirst reference image, selects the qth (q is an integer that satisfiesp+2≤q≤N−1) image as the second reference image, and selects the rth (ris an integer that satisfies p+1≤r≤q−1) image as the determinationtarget image. The processing section 100 determines whether or not thedetermination target image can be deleted based on the results of theprocess based on the deformation information about the first referenceimage and the determination target image, and the deformationinformation about the second reference image and the determinationtarget image, and the process that utilizes the structural element thatcorresponds to the attention area. When it has been determined that the(p+1)th to (q−1)th images can be deleted, the processing section 100selects the (q+1)th image as the second reference image.

This makes it possible to set the reference images to precede or followthe determination target image (see FIGS. 11A and 11B), and implementthe image summarization process based on the coverage ratio. Since tworeference images are used, it is likely that it is determined that thedetermination target image can be deleted, and the number of imagesincluded in the summary image sequence (the number of images after thesummarization process) can be reduced. When it has been determined thatthe determination target image that is set between the first referenceimage and the second reference image can be deleted (when all of thedetermination target images can be deleted in a narrow sense), it islikely that the determination target image that is set between the firstreference image and the second reference image can be deleted even whenthe interval between the first reference image and the second referenceimage is increased, and the second reference image is updated with theimage that follows the current second reference image.

According to the second embodiment, since the reference image can be setto follow the determination target image, the first image need notnecessarily be set to be the first reference image when the process isperformed for the first time. If all of the preceding images are coveredby a given image that follows the second image, the preceding images canbe deleted by setting the given image to be the reference image.

The processing section 100 may perform a process that allows the imageselected as the first reference image to be included in the summaryimage sequence. When it has been determined that at least one of the(p+1)th to (q−1)th images cannot be deleted, the processing section 100may set a partial image sequence that includes the (q−1)th to Nth imagesto be the input image sequence, and perform the process on the set inputimage sequence after setting the value p to 1.

This makes it possible to allow the first reference image to be includedin the summary image sequence in the same manner as in the firstembodiment in which the reference image is allowed to be included in thesummary image sequence. Since the case where at least one of thedetermination target images situated between the first reference imageand the second reference image cannot be deleted corresponds to the casewhere the interval between the first reference image and the secondreference image is increased to a large extent, it is necessary to allowthe image that precedes (immediately precedes in a narrow sense) thesecond reference image to remain in the summary image sequence.Therefore, a partial image sequence that includes the (q−1)th to Nthimages is set to be the input image sequence, and the process thatselects the first reference image, the second reference image, and thedetermination target image, the deletion determination process, thesecond reference image update process (optional), and the like areperformed on the set input image sequence. Since it is necessary toallow the first image of the partial image sequence to remain in thesummary image sequence, it is desirable to set the parameter p to 1.

3.2 Modification (Another Second Reference Image Update Method)

A modification of the second embodiment is described below. In thismodification, a similar expression is repeatedly used in connection withthe deletion determination process when describing the second referenceimage selection method. The expression “the qth image is OK” is usedwhen the qth image has been selected as the second reference image, andit has been determined by the deletion determination process that all ofthe images situated between the first reference image and the secondreference image can be deleted, and the expression “the qth image is NG”is used when the qth image has been selected as the second referenceimage, and it has been determined by the deletion determination processthat at least one of the images situated between the first referenceimage and the second reference image cannot be deleted, for convenienceof description.

According to the above method, the second reference image is selectedagain when the qth image is OK, and the second reference image to beselected is limited to the (q+1)th image.

When the first to Nth images have been input as the input imagesequence, the first image has been selected as the first referenceimage, and the qth image has been selected as the second referenceimage, q−2 images (second to (q−1)th images) are candidates for thedetermination target image, and the determination process is performedq−2 times. If the image summarization process has ended withoutdetermining that the determination target image cannot be deleted, avalue within the range from 3 to N (N+1 may be included in the rangewhen a virtual image is taken into account) is selected as q, and theprocess must be performed at least 1+2+3+ . . . +(N−2)=(N−2)(N−1)/2times (i.e., the amount of calculations is N²). Specifically, the abovemethod has a disadvantage in that the amount of calculationssignificantly increases when N is very large.

The amount of calculations can be reduced by increasing the selectionrange when selecting the second reference image again instead oflimiting the selection target to the adjacent image. Specifically, whenthe qth image is OK, the second reference image is not limited to the(q+1)th image, but is allowed to be selected from the (q+2)th image andthe subsequent images. In this case, even when the qth image is NG, itis unknown whether the (q−1)th image is OK since the (q−1)th image maynot have been selected as the second reference image. Therefore, thedetermination process is basically performed on the preceding image byselecting the image that precedes the qth image as the second referenceimage instead of necessarily allowing the (q−1)th image to remain in thesummary image sequence when the qth image is NG as described above inconnection with the basic method according to the second embodiment.

In this modification, the next summary image that follows the firstreference image is searched by updating the second reference image withthe subsequent image when the qth image is OK, and updating the secondreference image with the preceding image when the qth image is NG untilthe end condition is satisfied. The number of images selected as thesecond reference image until the next summary image is found can bereduced by appropriately setting the position of the second referenceimage, and the amount of calculations can be reduced. Note that theamount of calculations when using the above method may be smaller thanthe amount of calculations when using this modification depending on theposition of the next summary image that follows the first referenceimage. The method according to this modification is described in detailbelow.

A system configuration example of the image processing device is thesame as that illustrated in FIG. 8. The second reference image selectionprocess (update process) performed by the second reference imageselection section 1007 differs from that described above. The followingdescription focuses on this difference, and detailed description of thesame features is omitted.

When the input image sequence has been input, the reference imageselection section 1001 selects the first reference image. For example,the reference image selection section 1001 selects the first image ofthe input image sequence as the first reference image. When the inputimage sequence is the image sequence acquired by the image sequenceacquisition section 200 (i.e., when the first reference image selectionprocess is performed for the first time), an image other than the firstimage (e.g., a virtual zeroth image) may be selected as the firstreference image. The following description is given on the assumptionthat the first reference image is the first image unless otherwisespecified.

The second reference image is then selected. Specifically, a secondreference image selection interval is set that corresponds to the imagesto be selected as the second reference image (corresponding to the rangein which the next summary image that follows the first reference imageis searched). A semi-open interval [i, j) corresponding to the ith tojth images is set to be the second reference image selection interval. icorresponds to the image that immediately follows the first referenceimage (i=2 in a narrow sense), and j is set to N+2. j is set to N+2since a virtual (N+1)th image can be set to be the second referenceimage in the same manner as in the case of setting a virtual zerothimage to be the first reference image. A case where the second referenceimage is the (N+1)th image corresponds to the case where whether or notall of the images that follow the first reference image can be coveredonly by the first reference image, and the second reference image isunnecessary is determined.

The second reference image is selected from the second reference imageselection interval. The second reference image is determined based on agiven condition in order to efficiently perform the process.Specifically, when the second reference image is selected for the firsttime after the first reference image has been set, the (i+1)th image(third image in a narrow sense) is selected as the second referenceimage in the same manner as in the basic method according to the secondembodiment.

FIG. 13A illustrates the above process. FIG. 13A illustrates an imagesequence in which N=12. The first reference image is the first image,the second reference image selection interval corresponds to the secondto fourteenth images (i=2, j=14), and the second reference image is thethird image.

After the second reference image has been selected, the determinationtarget image selection process, the coverage area (non-coverage area)calculation process, the attention area miss probability determinationprocess, and the deletion determination process are performed (repeated)in the same manner as described above (detailed description thereof isomitted).

When a given image (the third image during the first process) has beenselected as the second reference image, and the given image is OK (i.e.,the position of the second reference image can be further shifted awayfrom the first reference image), the image that follows the currentsecond reference image is selected as the next second reference image inthe same manner as in the basic method according to the secondembodiment. Note that the second reference image may be shifted by twoor more images instead of shifting the second reference image by oneimage.

For example, when the current second reference image is the ath imagefrom the first reference image, the (2×a)th image from the firstreference image may be selected as the next second reference image.Specifically, when the third image (i.e., the second image from thefirst reference image) has been selected as the second reference image,and the third image is OK, the fifth image (i.e., the fourth image fromthe first reference image) is selected as the next second referenceimage (see FIG. 13B).

When the qth image is OK, it is unnecessary to allow the (q−1)th imageand the images the precede the (q−1)th image to remain as the summaryimage. In this case, the second reference image selection interval maybe updated since no advantage is obtained even if the image thatprecedes the qth image is selected as the second reference image.Specifically, the starting point i of the second reference imageselection interval may be set to i=q. Since the second reference imageis selected from the second reference image selection interval, theimage that precedes the current second reference image is not selectedwhen the starting point i is set to i=q. For example, when the thirdimage is OK (i.e., when the second image is not selected as the summaryimage), the second image is excluded from the second reference imageselection interval, and the starting point of the second reference imageselection interval is updated with the third image (see FIG. 13B).

Likewise, when the fifth image is OK, the ninth image is selected as thenext second reference image, and the starting point of the secondreference image selection interval is updated with the fifth image (seeFIG. 13C).

However, when the qth image has been selected as the second referenceimage, and the qth image is OK, it is likely that an image that issituated significantly away from the qth image is selected as the nextsecond reference image as the value q increases (see FIG. 13C). Forexample, a situation may occur in which an image that follows the(N+1)th image may be set to be a candidate for the second referenceimage (i.e., the second reference image cannot be selected), or theinterval between the current second reference image and the next secondreference image increases to a large extent, and the next summary imagesearch process becomes inefficient.

Therefore, another method may be used in combination when selecting animage that follows the current second reference image as the next secondreference image. For example, the next second reference image may bedetermined based on the value (q+j)/2. For example, when the ninth imageis OK, the starting point of the second reference image selectioninterval is updated with the ninth image (i.e., the second referenceimage selection interval is a semi-open interval [9, 14)). Specifically,the center of the search range can be set to be the processing target byselecting an image around the center of the search range as the nextsecond reference image. The method that halves the search range bydetermining the center of the search range is a widely known binarysearch method, and it is known that the binary search method isadvantageous from the viewpoint of the amount of calculations. Thebinary search method can be applied to the second reference imageselection interval since all of the images that precede a given imageare determined to be OK when the given image is OK, and all of theimages that follow a given image are determined to be NG when the givenimage is NG. Specifically, it is considered that an efficient processcan be implemented by selecting the next second reference image from thecenter point between the current second reference image and the endpoint of the second reference image selection interval.

A method that doubles the distance from the first reference image, and amethod that corresponds to the binary search method may be used incombination. For example, when the qth image is the current secondreference image, the kth image that satisfies the following expression(1) may be selected as the next second reference image. Note that min(a,b) outputs the smaller of a and b.

$\begin{matrix}{k = {\min\left( {{{2q} - 1},\frac{q + j}{2}} \right)}} & (1)\end{matrix}$

When the qth image is NG, the next second reference image is selectedfrom the images that precede the current second reference image. Thenext second reference image may be determined using various methods. Forexample, the next second reference image may be determined using amethod that corresponds to the binary search method. In this case, sincethe starting point of the second reference image selection interval isthe ith image, the next second reference image is determined based onthe value (i+q)/2. Since the qth image is NG, the qth image and theimages that follow the qth image are not selected as the secondreference image. Therefore, the end point of the second reference imageselection interval can be updated (i.e., j=q). FIG. 13D illustrates anexample when the ninth image is NG. The seventh image is selected as thenext second reference image, and the end point j of the second referenceimage selection interval is updated with j=9.

Note that a semi-open interval is used as the second reference imageselection interval for convenience of explanation. Specifically, sincethe qth image may be selected as the summary image when the qth image isOK, it is desirable that the starting point i (i=q) of the secondreference image selection interval be included in the second referenceimage selection interval. Since the qth image is not selected as thesummary image when the qth image is NG, it is desirable that the endpoint j (j=q) of the second reference image selection interval not beincluded in the second reference image selection interval. Therefore,the second reference image selection interval is represented by [i, j).The second reference image selection interval may be represented by anopen interval or a closed interval depending on the sign or theexpression.

The second reference image selection interval (i.e., the next summaryimage search range in a narrow sense) is reduced by the above process.Since the next summary image is the kth image when the kth image is OKand the (k+1)th image is NG, the process is terminated when an imagethat is OK and an image that is NG are adjacent to each other. In theabove example, it is considered that the process is performed in abinary search manner (see FIG. 13E). In FIG. 13E, the ith image is OK,the jth image is NG, and the qth image between the ith image and the jthimage is selected as the second reference image. FIG. 13F illustratesthe case where the qth image is OK, and FIG. 13G illustrates the casewhere the qth image is NG. In FIGS. 13F and 13G, the starting point andthe end point of the second reference image selection interval areadjacent to each other, the image corresponding to the starting point isOK, and the image corresponding to the end point is NG. In this case,the image corresponding to the starting point is selected as the nextsummary image, and the search process on the input image sequence isterminated.

When the next summary image has been found, a partial image sequencethat includes the next summary image and the images that follow the nextsummary image is set to be the input image sequence in the same manneras in the case of using the basic method according to the secondembodiment. Therefore, the partial image sequence setting section 1008sets the starting point of the second reference image selection intervaland the subsequent images to be the partial image sequence, and sets thepartial image sequence to be the next input image sequence. Thesubsequent process is the same as described above, and detaileddescription thereof is omitted.

FIG. 14 is a flowchart illustrating the above process. Note that stepsS301 to S303 are the same as the steps S201 to S203, respectively. Afterthe first reference image has been selected in the step S303, the secondreference image selection interval is set (S311). When the step S311 isperformed immediately after the step S303, a semi-open interval [i, j)that satisfies i=2 and j=N+2 may be set, for example. When the step S311is performed after the step S304 or S308, the second reference imageselection interval is updated.

When the second reference image selection interval has been set (orupdated) in the step S311, whether or not the starting point and the endpoint of the second reference image selection interval are adjacent toeach other (i.e., whether or not j=i+1 is satisfied) is determined(S312). When it has been determined that the starting point and the endpoint of the second reference image selection interval are adjacent toeach other in the step S312 (i.e., when it has been determined that theith image is the next summary image that follows the first image (seeFIG. 13F)), the ith image and the subsequent images are set to be thepartial image sequence (S309), and the process in the step S303 isperformed.

When it has been determined that the starting point and the end point ofthe second reference image selection interval are not adjacent to eachother in the step S312 (i.e., when the next summary image has not beenfound), the second reference image is selected from the second referenceimage selection interval set in the step S311 (S310). When the processin the step S310 is performed for the first time after the firstreference image has been set in the step S303, the (i+1)th image (i.e.,the image that follows the first reference image at an interval of oneimage) may be selected, for example. When the process in the step S310is not performed for the first time after the first reference image hasbeen set in the step S303, the next second reference image is selectedcorresponding to the position of the current second reference image.

When the second reference image has been selected in the step S310, thedetermination target image is selected (S304). The deformationinformation acquisition process (S305), the coverage area calculationprocess (S306), the attention area miss probability determinationprocess (S307), and the image deletion determination process (S308)after the determination target image has been selected are performed inthe same manner as in the steps S205 to S208, respectively. When it hasbeen determined that the determination target image can be deleted inthe step S308, the determination target image is updated with the imagethat immediately follows the current determination target image (S304),and the process is performed in the same manner as described above.Whether or not all of the images situated between the first referenceimage and the second reference image can be deleted, or at least one ofthe images situated between the first reference image and the secondreference image cannot be deleted, is determined by repeating theprocesses in the steps S304 to S308. When it has been determined thatall of the images situated between the first reference image and thesecond reference image can be deleted (determination target image=secondreference image), the process in the step S311 is performed. When it hasbeen determined that at least one of the images situated between thefirst reference image and the second reference image cannot be deleted,it is determined that the determination target image cannot be deletedin the step S308, and the process in the step S311 is performed. It isnecessary to store information that indicates whether the step S311 isperformed after the step S304 or S308, and change the process in thestep S311 based on the information (not illustrated in FIG. 14).

When the step S311 is performed after the step S304 (i.e., when all ofthe images can be deleted), the starting point of the second referenceimage selection interval is updated, and the image that follows thecurrent second reference image is selected as the next second referenceimage in the step S310. When the step S311 is performed after the stepS308 (i.e., when at least one of the images cannot be deleted), the endpoint of the second reference image selection interval is updated, andthe image that precedes the current second reference image is selectedas the next second reference image in the step S310.

According to this modification, when the pth image is selected from theinput image sequence that includes the first to Nth images as the firstreference image, and the qth image is selected as the second referenceimage, the processing section 100 selects the second reference imagefrom the second reference image selection interval in which the startingpoint and the end point are set corresponding to the (p+2)th image andthe Nth image. The processing section 100 determines whether or not thedetermination target image can be deleted based on the results of theprocess that utilizes the deformation information about the firstreference image and the determination target image, the process thatutilizes the deformation information about the second reference imageand the determination target image, and the process that utilizes thestructural element that corresponds to the attention area. When it hasbeen determined that the (p+1)th to (q−1)th images can be deleted, theprocessing section 100 selects the xth (x is an integer that satisfiesx>q) image included in the second reference image selection interval asthe next second reference image. In this case, the processing section100 may update the starting point of the second reference imageselection interval with the qth image.

The second reference image selection interval includes the (p+2)th toNth images that are candidates for the second reference image. However,since a virtual image (e.g., (N+1)th image) can be selected as thesecond reference image, the end point of the second reference imageselection interval may be larger than N. Since the second referenceimage selection interval is used as the next summary image search range,an image that is not selected as the second reference image, but may beselected as the summary image may be included in the second referenceimage selection interval. In this case, the image ((p+1)th image) thatimmediately follows the first reference image may be set to be thestarting point of the second reference image selection interval.

This makes it possible to flexibly determine the position of the nextsecond reference image when updating the second reference image. Sincethe basic method according to the second embodiment reduces the searchrange by thoroughly checking the search range from the first image, theamount of calculations may significantly increase depending on theposition of the second reference image. In contrast, the search rangecan be significantly reduced by the unit determination that determineswhether the qth image is OK or NG by allowing a non-adjacent image to beselected as the next second reference image. This makes it possible toreduce the amount of calculations, and reduce the load imposed on thesystem, or reduce the processing time.

When it has been determined that at least one of the (p+1)th to (q−1)thimages cannot be deleted, the processing section 100 may select the yth(y is an integer that satisfies y<q) image included in the secondreference image selection interval as the next second reference image.In this case, the processing section 100 updates the end point of thesecond reference image selection interval with the qth image.

This makes it possible to select the image that precedes the currentsecond reference image as the next second reference image when updatingthe second reference image. Since the search process is not limited to aprocess that selects the adjacent image, the range that precedes thecurrent second reference image may not have been searched, and mayinclude a correct answer depending on the deletion determination result.In this case, it is possible to perform an appropriate process byperforming a forward search process. Moreover, the next second referenceimage need not be selected from the adjacent image.

When the jth (j is an integer) image corresponds to the end point of thesecond reference image selection interval, the processing section 100may set the value x based on the value (q+j)/2. Alternatively, when theith (i is an integer) image corresponds to the starting point of thesecond reference image selection interval, the processing section 100may set the value y based on the value (i+q)/2.

This makes it possible to use the binary search method when selectingthe next second reference image. The image that is situated between thecurrent second reference image and the end point is selected whenperforming a backward search process, and the image that is situatedbetween the current second reference image and the starting point isselected when performing a forward search process. This makes itpossible to halve the search range (corresponding to the length of thesecond reference image selection interval). It is expected that theentire search ranges is completely searched when log N images areselected as the second reference image. Therefore, the amount ofcalculations can be reduced to N×log N. When N is very large, the amountof calculations can be significantly reduced as compared with the basicmethod according to the second embodiment (the amount of calculations isN²). Note that the value (q+j)/2 and the value (i+q)/2 are notnecessarily an integer, and an image corresponding to each value may beabsent. In such a case, the maximum integer that does not exceed thevalue (q+j)/2, or an integer that is larger than the value (q+j)/2 by 1may be used, for example.

The processing section 100 may perform a process that allows the imageselected as the first reference image to be included in the summaryimage sequence when the starting point and the end point of the secondreference image selection interval are adjacent to each other as aresult of updating the starting point or the end point of the secondreference image selection interval. The processing section 100 may setthe partial image sequence that includes the image corresponding to thestarting point and the images that follow the image corresponding to thestarting point in the input image sequence, to be the input imagesequence, and perform the process on the set input image sequence aftersetting the value p to 1.

The expression “the starting point and the end point of the secondreference image selection interval are adjacent to each other” meansthat the image corresponding to the starting point and the imagecorresponding to the end point are adjacent to each other in the inputimage sequence. When N images have been set to be the input imagesequence, it is considered that the input image sequence is a set oftemporally or spatially continuous images. Therefore, the positionwithin the image sequence can be defined based on the continuity. Forexample, an image acquired at an earlier time precedes an image acquiredat a later time. Specifically, the images included in the input imagesequence are referred as first to Nth images, and it is determined thatan image is situated at a forward position when the number assigned tothe image is small. Therefore, when the ith image and the jth (>i) mageincluded in the image sequence are adjacent to each other, j=i+1 issatisfied.

This makes it possible to set a condition based on the starting pointand the end point (or the length) of the second reference imageselection interval as a condition whereby the process on the input imagesequence is terminated. An image among the images that are determined tobe OK when selected as the second reference image that is expected to besituated farthest from the first reference image can be selected as thefirst image (corresponding to the next summary image) of the partialimage sequence by setting the termination condition. This is because thetermination condition is equivalent to the condition whereby theposition at which the image that is OK and the image that is NG areadjacent to each other is searched (see FIG. 13F, for example). Thismakes it possible to reduce the number of summary images included in thesummary image sequence that is finally output, and reduce the burdenimposed on the user, for example.

3.3 Modification (Second Reference Image Initial Setting)

According to the second embodiment and the modification thereof, whenthe input image sequence (that may be the image sequence acquired by theimage sequence acquisition section 200, or may be the partial imagesequence that is part of the image sequence acquired by the imagesequence acquisition section 200) has been input, the second referenceimage that is set first, is limited to the image that follows the firstreference image at an interval of one image.

Note that the initial position of the second reference image may differfrom the above position. For example, it is not likely that an intervalin which similar images continue and an interval in which the number ofsimilar images is small are adjacent to each other in the actual imagesequence. Specifically, it is considered that the length of the nextsummary interval (that indicates the distance between the adjacentsummary images) is close to the length of the preceding summaryinterval. Therefore, when a plurality of summary images have beenobtained, and information that corresponds to the length of thepreceding summary interval has been acquired, it is expected that acorrect answer is obtained more quickly, and the amount of calculationscan be reduced by setting the initial position of the second referenceimage to a position that is situated away from the first reference imageby the length of the preceding summary interval.

Specifically, the length g of the summary interval is acquired from thepreceding summary image and the summary image that immediately precedesthe preceding summary image. When the second reference image selectioninterval is j), the second reference image is set to the (i+g)th imageinstead of the (i+1)th image. Note that the length g of the summaryinterval cannot be acquired when the number of summary images is 0 or 1.In this case, the initial position of the second reference image is setwithout using the length g of the summary interval. For example, whenthe number of summary images is 0, the (i+1)th image may be selected asthe second reference image. When the number of summary images is 1, the(i+g′)th image may be selected as the first second reference image (g′is the length from the first image of the image sequence acquired by theimage sequence acquisition section 200 to the summary image).

The second reference image may be updated in various ways. For example,the next second reference image may be selected using the binary searchmethod (see above).

Since it is likely that the next summary image is present around the(i+g)th image, the number of searches until the next summary image isfound may increase when the updated second reference image is situatedaway from the (i+g)th image. In such a case, the image that is adjacentto the preceding second reference image may be selected as the nextsecond reference image, as described above in connection with the basicmethod according to the second embodiment. However, since thedetermination process is not performed on the (i+1)th to (i+g−1)thimages, the next summary image may be present within this range.Therefore, the second reference image need not necessarily be updatedone by one in the backward direction, but may be updated one by one inthe forward direction depending on the deletion determination result.

4. Third Embodiment

A modification of the process that is performed after the referenceimage and the determination target image have been selected, anddetermines whether or not the determination target image can be deletedis described below. In the third embodiment, the coverage area and thenon-coverage area are not necessarily calculated. However, the deletiondetermination process is performed in the same manner as described abovebased on whether or not the entirety of the attention area is includedwithin the area (corresponding to the non-coverage area) that is notcovered by the reference image.

FIG. 15 illustrates a system configuration example of an imageprocessing device according to the third embodiment. In FIG. 15, theprocessing section 100 further includes a coverage-requiring areageneration section 1011 and a coverage-requiring area inclusiondetermination section 1012.

The coverage-requiring area generation section 1011 generates acoverage-requiring area based on the determination target image and thestructural element. The coverage-requiring area inclusion determinationsection 1012 determines whether or not the generated coverage-requiringarea is included in the reference image. The details of the processperformed by the coverage-requiring area generation section 1011 and theprocess performed by the coverage-requiring area inclusion determinationsection 1012 are described later. The attention area miss probabilitydetermination section 1013 according to the third embodiment determinesthe probability that the attention area is missed based on the inclusionrelationship between the coverage-requiring area and the referenceimage.

The details of the process are described below. In the third embodiment,the erosion process that utilizes the structural element is performed onthe entire determination target image to determine whether or not thedetermination target image can be deleted. As illustrated in FIGS. 16Aand 16B, the erosion process that utilizes the structural element thatcorresponds to the attention area is performed on the determinationtarget image, and the residual area is set to be the coverage-requiringarea. When the structural element has a circular shape, a circle isdrawn around a point at the boundary (outer edge) of the determinationtarget image, and the overlapping area of the circle and thedetermination target image is deleted to determine the residual area tobe the coverage-requiring area.

In the third embodiment, an area that is deleted by the erosion processis set to satisfy the condition whereby the entirety of the attentionarea is included within the area. In this case, since the entirety ofthe attention area is not included within the deleted area, the deletedarea need not be covered by the reference image. In other words, thecoverage-requiring area that has not been deleted by the erosion processis an area that is required to be covered by the reference image.

Specifically, a comparison process is performed on the reference imageand the coverage-requiring area based on the deformation informationabout the reference image and the determination target image. It isdetermined that the determination target image can be deleted when thecoverage-requiring area is covered by the reference image, and it isdetermined that the determination target image cannot be deleted whenthe coverage-requiring area is not covered by the reference image.

It is desirable that an area that is deleted by the erosion process bethe largest area that satisfies the above condition. When no area isdeleted by the erosion process (e.g., when the erosion process is notperformed, or an ideal point having no size is set to be the structuralelement), the condition whereby the entirety of the attention area isnot included within the deleted area is satisfied. However, since theentire determination target image is set to be the coverage-requiringarea, it is determined that the determination target image can bedeleted only when the entire determination target image is covered bythe reference image (when the coverage ratio is 100%). Therefore, theeffect of reducing the number of images through the image summarizationprocess significantly decreases, and it is difficult to achieve theoriginal object of the image summarization process. Specifically, it isnecessary to increase the probability that it is determined that thedetermination target image can be deleted in order to effectively reducethe number of images. In the third embodiment, it is desirable todetermine that the determination target image can be deleted with highprobability (necessarily in a narrow sense) when the condition wherebyat least part of the attention area is captured within the referenceimage even if the determination target image is deleted is satisfied.Therefore, it is desirable that an area that is deleted by the erosionprocess be the largest area that satisfies the above condition. This isbasically implemented by increasing the size of the structural elementas much as possible.

In the first embodiment, the erosion process that utilizes thestructural element is performed on the non-coverage area, and whether ornot the entirety of the area that corresponds to the attention area isincluded within the non-coverage area is determined based on thepresence or absence of the residual area. In the third embodiment, theerosion process is performed on the entire determination target image,and the entirety of the attention area is necessarily included withinthe determination target image during normal observation. Note that sucha situation may not occur depending on the imaging magnification or thelike. Therefore, the residual area necessarily occurs, and thedetermination process based on the presence or absence of the residualarea is meaningless. The above difference in process occurs depending onwhether the erosion process is performed on the area for which inclusionof the entirety of the attention area is determined (first embodiment),or a situation is prevented in which the entirety of the attention areais included within the area deleted by the erosion process (thirdembodiment).

Accordingly, the third embodiment differs from the first embodiment asto the structural element setting method based on the attention area. Inthe first embodiment, a shape change process that sets a rectangularstructural element from a circular attention area may be performed, forexample. However, the structural element is basically set to have a sizesimilar to that of the attention area (the same size and the same shapeas those of the attention area in a narrow sense). However, when thestructural element is a circle having a radius r, and the center of thecircle is set to be the reference point, an area that is deleted by theerosion process is an area within the distance r from the boundary ofthe area subjected to the erosion process (i.e., the boundary of thedetermination target image) (see FIG. 16B). Specifically, when theattention area is a circle having a radius R (R=r), the area illustratedin FIG. 16B that is deleted by the erosion process satisfies thecondition whereby the entirety of the attention area is not includedwithin the deleted area, but has an area smaller than the largest area(i.e., the above condition is satisfied even when a larger area isdeleted). Therefore, the effect of reducing the number of images throughthe image summarization process decreases when the structural element isset in the same manner as in the first embodiment.

In the third embodiment, it is desirable to set the structural elementto have a size larger than that of the attention area, differing fromthe first embodiment. For example, when the structural element is acircle having a radius r, and the center of the circle is set to be thereference point, a range within the distance r from the boundary of thedetermination target image is deleted by the erosion process. Therefore,the radius of a circle that is entirely included within the deletedrange is r/2. Specifically, when the attention area is a circle having aradius R, the entirety of the attention area is not included within thedeleted area (range) when R≥r/2. Therefore, the radius r of thestructural element is r≤2R. For example, a circle having a radius r=2Rmay be used as the structural element. Likewise, when the attention areais a L×L square, a 2L×2L square may be used as the structural element.Specifically, the structural element may be basically set to have adiameter or a maximum size twice larger than that of the attention area.

The accuracy of the deletion determination process may decrease (i.e.,it is determined that the determination target image can be deletedalthough the attention area may be missed) when using the above settingdepending on the shape of the determination target image or thestructural element. In such a case, it is necessary to set the size ofthe structural element corresponding to the shape of the determinationtarget image or the like in order to increase the accuracy of thedeletion determination process, and the setting process may increase theprocessing load. Therefore, when it is desired to use the methodaccording to the third embodiment so that the processing load decreases(at the expense of a decrease in accuracy) as compared with the methodaccording to the first embodiment, the structural element may be set tohave a size twice larger than that of the attention area independentlyof the shape of the determination target image or the like.

Note that the size of the structural element may be appropriatelydetermined corresponding to the shape of the determination target imageor the like. A specific example of such a case is described below withreference to FIGS. 16A to 16C. When the erosion process is performed ona quadrangular determination target image using a circular structuralelement having a radius r, the peripheral area of the determinationtarget image is deleted (see FIGS. 16A and 16B). In this case, theradius of the largest circle that is included within the deleted areabecomes a maximum at each corner of the determination target image (seeFIG. 16C), and the following expression (2) must be satisfied.√{square root over (2)}r≤R+√{square root over (2)}R  (2)

Specifically, when the attention area is a circle having a radius R, theradius r of a circle used as the structural element must satisfy thefollowing expression (3). The value r when an equal sign is satisfiedmay be used as the radius of the structural element.

$\begin{matrix}{r \leq {\left( {1 + \frac{\sqrt{2}}{2}} \right)R}} & (3)\end{matrix}$

The comparison process on the reference image and the coverage-requiringarea may be implemented by the deformation process based on thedeformation information in the same manner as the process according tothe first embodiment that calculates the coverage area. As illustratedin FIG. 17A, the coverage-requiring area may be deformed, and projectedonto the reference image, for example. In this case, thecoverage-requiring area is covered by the reference image when thedeformed coverage-requiring area is included within the reference image.In this case, the determination target image can be deleted. Thecoverage-requiring area is not covered by the reference image when atleast part of the deformed coverage-requiring area is not includedwithin the reference image. In this case, the determination target imagecannot be deleted.

As illustrated in FIG. 17B, the comparison process on the referenceimage and the coverage-requiring area may be implemented by deformingthe reference image. Specifically, the reference image is deformed basedon the deformation information, and projected onto the determinationtarget image. The deformed reference image corresponds to the coveragearea described above in connection with the first embodiment. Thedetermination target image can be deleted when the coverage-requiringarea is included within the coverage area, and cannot be deleted when atleast part of the coverage-requiring area is not included within thecoverage area.

FIG. 18 is a flowchart illustrating the process according to the thirdembodiment. Note that steps S401 to S405 are respectively the same asthe steps S101 to S105 illustrated in FIG. 5, and detailed descriptionthereof is omitted. After the step S405, the coverage-requiring area isgenerated based on the structural element generated in the step S402(S410). Specifically, the erosion process that utilizes the structuralelement is performed on the determination target image (see FIGS. 16Aand 16B), and the residual area is set to be the coverage-requiringarea. Whether or not the coverage-requiring area is included within thereference image is determined (S411). Specifically, thecoverage-requiring area may be deformed based on the deformationinformation, and compared with the reference image (see FIG. 17A), orthe reference image may be deformed based on the deformationinformation, and compared with the coverage-requiring area (see FIG.17B).

The probability that the attention area is missed, is determined basedon whether or not it has been determined that the coverage-requiringarea is included within the reference image in the step S411 (attentionarea miss probability determination process (S407)). Note that stepsS408 and S409 are the same as the steps S108 and S109 illustrated inFIG. 5.

According to the third embodiment, the processing section 100 mayperform the erosion process that utilizes the structural element on thedetermination target image to calculate the coverage-requiring area asthe process that utilizes the structural element. The processing section100 may determine whether or not the determination target image can bedeleted based on the calculated coverage-requiring area and thedeformation information.

This makes it possible to simply determine the probability that theattention area is missed. Since it is considered that the size and theshape of the plurality of images included in the image sequence rarelychange (all of the plurality of images have the same size and the sameshape in a narrow sense), it is considered that the determination targetimage normally has the same size and the same shape. When the erosionprocess that utilizes the same structural element is performed on aplurality of images (areas) having the same size and the same shape, anidentical area is obtained by the erosion process. Specifically, when aplurality of determination target images have the same size and the sameshape, the coverage-requiring area calculated from each determinationtarget image is identical, and it is not likely that it is necessary toperform the erosion process that utilizes the structural element on allof the determination target images. Therefore, when the process thatcalculates the coverage-requiring area has been performed once, theresults can be used for the determination target image having the samesize and the same shape. This makes it possible to reduce the processingload. Since it is considered that the determination target image has asimple shape (e.g., quadrangular shape or circular shape), the processis facilitated as compared with the erosion process according to thefirst embodiment that is performed on the non-coverage area. The size ofthe structural element may be set to be twice the size of the attentionarea. Note that the attention area may be missed when r=2R (see FIG.16C). Therefore, it is desirable to change the process corresponding tothe situation. When it is desired to reduce the processing load even ifthe attention area may be missed to some extent, the processing load canbe further reduced by setting the size of the structural element to betwice the size of the attention area independently of the shape of thedetermination target image or the attention area. When it is desired toprevent a situation in which the attention area is missed, the ratio ofthe size of the structural element to the size of the attention area maybe determined corresponding to the shape of the determination targetimage or the attention area.

The processing section 100 may perform a process that deforms thecoverage-requiring area based on the deformation information about thereference image and the determination target image as the process thatutilizes the deformation information. The processing section 100 maydetermine that the determination target image can be deleted when thedeformed coverage-requiring area is included within the reference image.

The processing section 100 may perform a process that deforms thereference image based on the deformation information about the referenceimage and the determination target image as the process that utilizesthe deformation information. The processing section 100 may determinethat the determination target image can be deleted when thecoverage-requiring area is included within the deformed reference image.

This makes it possible to implement the inclusion determination processillustrated in FIG. 17A or 17B. The coverage-requiring area is an areathat is required to be covered by the reference image. Since thedetermination target image (and the coverage-requiring area) is normallydeformed with respect to the reference image, the comparison process onthe reference image and the coverage-requiring area must deform at leastone of the reference image and the coverage-requiring area based on thedeformation information.

When an intermediate image is included in the input image sequencebetween the reference image and the determination target image, thereference image may be deformed based on the deformation informationabout the reference image and the intermediate image, thecoverage-requiring area may be deformed based on the deformationinformation about the determination target image and the intermediateimage, and the inclusion determination process may be performed on thedeformed reference image and the deformed coverage-requiring area todetermine whether or not the determination target image can be deleted.Note that it is desirable to deform one of the reference image and thecoverage-requiring area taking account of the processing load and thelike.

5. Fourth Embodiment

Whether or not the determination target image can be deleted (i.e.,whether or not the entirety of the attention area is included within thenon-coverage area) may be determined without performing the erosionprocess that utilizes the structural element. In this case, thereference image is deformed based on the deformation information tocalculate the coverage area in the same manner as in the firstembodiment.

A system configuration example of an image processing device accordingto the fourth embodiment is the same as that illustrated in FIG. 4 (seethe first embodiment), and detailed description thereof is omitted. Thedetails of the process are the same as those illustrated in FIG. 5, anddetailed description thereof is omitted. The fourth embodiment differsfrom the first embodiment as to the process performed by the attentionarea miss probability determination section 1013 illustrated in FIG. 4,and the process performed in the step S107 illustrated in FIG. 5.

In the fourth embodiment, a plurality of points are set at the boundaryof the coverage area, and the minimum value of the distance from eachpoint to the boundary of the determination target image is calculated.For example, when the coverage area and the determination target imageillustrated in FIG. 19A have been acquired, the minimum valuecorresponding to a point p1 is k1, and the minimum values correspondingto points p2 to p4 are k2 to k4, respectively. Although FIG. 19Aillustrates the four points p1 to p4, the minimum value may becalculated corresponding to N points sufficient to ensure accuracy(corresponding to each pixel that corresponds to the boundary of thecoverage area in a narrow sense).

When N minimum values k1 to kN have been acquired corresponding to thepoints p1 to pN, the maximum value among the minimum values k1 to kN iscalculated. Since the maximum value among the minimum values k1 to kN isconsidered to be information that corresponds to the maximum size of thenon-coverage area, whether or not the entirety of the attention area isincluded within the non-coverage area can be simply determined byperforming the comparison process on the maximum value and the minimumsize of the structural element (having the same size as that of theattention area). Specifically, when the maximum value among the minimumvalues k1 to kN is larger than the minimum size of the structuralelement, it is determined that the entirety of the attention area may beincluded within the non-coverage area, and the determination targetimage cannot be deleted. When the maximum value among the minimum valuesk1 to kN is equal to or smaller than the minimum size of the structuralelement, it is determined that the entirety of the attention area is notincluded within the non-coverage area, and the determination targetimage can be deleted.

When the determination target image has a quadrangular shape, a largenon-coverage area may occur at each corner of the determination targetimage. In FIG. 19A, the minimum value corresponding to the point p4 isk4. However, it is appropriate to determine that the maximum size of thenon-coverage area is k′ (see the dotted line). Therefore, a plurality ofpoints may be set at the boundary of the determination target image, theminimum value of the distance from each point to the boundary of thecoverage area may be calculated, and the maximum value among a pluralityof calculated minimum values may be used in combination. This makes itpossible to appropriately calculate the information that corresponds tothe maximum size of the non-coverage area by utilizing the value k′.

When the determination target image has a circular shape (see FIG. 19B),it is considered that the minimum value corresponding to each point(e.g., the minimum values k5 and k6 corresponding to points p5 and p6)appropriately represents the minimum size of the non-coverage area.Specifically, when using a circular determination target image, itsuffices to use one of the method that sets a plurality of points at theboundary of the coverage area and the method that sets a plurality ofpoints at the boundary of the determination target image, differing fromthe case of using a quadrangular determination target image (see FIG.19A). Therefore, it is considered that the method according to thefourth embodiment can reduce the processing load when performing theprocess on a circular determination target image as compared with thecase performing the process on a quadrangular determination target imagewhile achieving a similar degree of accuracy. It is considered that itis possible to achieve a certain accuracy when using a determinationtarget image having a polygonal shape having a number of vertices(octagonal shape or hexadecagonal shape) by utilizing one of the methodthat sets a plurality of points at the boundary of the coverage area andthe method that sets a plurality of points at the boundary of thedetermination target image, although the accuracy may decrease ascompared with the case of using a circular determination target image.Note that the above two methods may be used in combination when thedetermination accuracy is important.

When using the method according to the fourth embodiment, the entiretyof the attention area may be included within the non-coverage area evenwhen the maximum value among the minimum values k1 to kN is larger thanthe minimum size of the structural element. For example, even when it isobvious that the entirety of the attention area is not included withinthe non-coverage area (see FIG. 19C), it is determined that thedetermination target image illustrated in FIG. 19C can be deleted sincea minimum value k7 corresponding to a point p7 is used for comparisonwith the minimum size of the structural element. Specifically, it may bedifficult to achieve the same degree of determination accuracy as thatachieved when using the erosion process according to the firstembodiment (e.g., the effect of reducing the number of images decreasesin FIG. 19C since the determination target image that can be deleted isallowed to remain) even when using a circular determination targetimage.

According to the fourth embodiment, the processing section 100 mayperform a process that deforms the reference image using the deformationinformation to calculate the coverage area (i.e., an area in which thedetermination target image is covered by the reference image (an area ofthe determination target image that is covered by the reference image)as the process that utilizes the deformation information. The processingsection 100 may perform a process that calculates the minimum distancefrom each point among a plurality of points that are set at the boundaryof the calculated coverage area to the outer edge of the determinationtarget image, and calculates the maximum value among the minimumdistances calculated corresponding to the plurality of points. Theprocessing section 100 may perform a process that acquires the minimumsize of the structural element, and determines whether or not thestructural element is included within the non-coverage area (i.e., anarea of the determination target image other than the coverage area)based on a comparison between the acquired minimum size and the maximumvalue among the minimum distances, as the process that utilizes thestructural element.

The minimum size of the structural element is a value that representsthe size of the structural element. For example, the minimum size refersto the diameter of a circular structural element, or the length of theshort side of a quadrangular structural element. When the attention areahas a complex shape, and it is difficult to calculate the minimum sizeof the attention area, it is possible to easily calculate the minimumsize of the structural element by setting the structural element to havea shape obtained by simplifying the shape of the attention area (e.g., ashape obtained by reducing the number of vertices of the shape of theattention area and circumscribed to the attention area).

This makes it possible to simply determine the probability that theattention area is missed. The minimum distance from each point at theboundary of the coverage area to the boundary (outer edge) of thedetermination target image can be easily calculated as illustrated inFIGS. 19A and 19B (depending on the shape of the determination targetimage). The minimum distance can be easily compared with the minimumsize of the structural element (having the same shape and the same sizeas those of the attention area). The method according to the fourthembodiment reduces the processing load as compared with the methodaccording to the first embodiment or the like.

The processing section 100 may perform a process that calculates theminimum distance from each point among a plurality of points that areset at the outer edge of the determination target image to the boundaryof the coverage area, and calculates the maximum value among the minimumdistances calculated corresponding to the plurality of points. Theprocessing section 100 may perform a process that acquires the minimumsize of the structural element, and determines whether or not thestructural element is included within the non-coverage area (i.e., anarea of the determination target image other than the coverage area)based on a comparison between the acquired minimum size and the maximumvalue among the minimum distances, as the process that utilizes thestructural element.

This makes it possible to perform a similar process by setting aplurality of points at the outer edge of the determination target image.Since the information that corresponds to the maximum size of thenon-coverage area is calculated from the distance between the outer edgeof the determination target image and the boundary of the coverage area,a plurality of points used to calculate the distance can be set toeither the determination target image or the coverage area. The methodthat sets a plurality of points at the boundary of the coverage area andthe method that sets a plurality of points at the boundary of thedetermination target image may be used either alone or in combination.For example, the method that sets a plurality of points at the boundaryof the coverage area and the method that sets a plurality of points atthe boundary of the determination target image may be used incombination when the determination accuracy is important, and one of themethod that sets a plurality of points at the boundary of the coveragearea and the method that sets a plurality of points at the boundary ofthe determination target image may be used when it is desired to reducethe processing load. Alternatively, whether or not to use the methodthat sets a plurality of points at the boundary of the coverage area andthe method that sets a plurality of points at the boundary of thedetermination target image in combination may be determined takingaccount of the shape (e.g., circular shape or quadrangular shape) of thedetermination target image.

6. Fifth Embodiment

A different deletion determination process that utilizes the structuralelement has been described above in connection with the firstembodiment, the third embodiment, and the fourth embodiment. The fifthembodiment illustrates a method that uses an arbitrary deletiondetermination process that utilizes the structural element and thedeletion determination process based on the coverage ratio.

FIG. 20 illustrates a configuration example of an image processingdevice according to the fifth embodiment. The image processing deviceaccording to the fifth embodiment differs from the image processingdevice illustrated in FIG. 4 in that the processing section 100 furtherincludes a coverage ratio calculation section 1004. The coverage ratiocalculation section 1004 calculates the coverage ratio based on thecoverage area calculated by the coverage area calculation section 1003,for example. The deletion determination section 1005 according to thefifth embodiment determines whether or not the determination targetimage can be deleted based on the coverage ratio calculated by thecoverage ratio calculation section 1004 and the probability (that theattention area is missed) that is determined by the attention area missprobability determination section 1013.

The details of the process are described below. The reference image isdeformed to calculate the coverage area in the same manner as in thefirst embodiment. The area ratio of the coverage area to thedetermination target image may be used as the coverage ratio. Note thatthe coverage ratio is information that represents the degree of coverageof the determination target image by the reference image, and is notlimited to the ratio and the like. For example, a plurality of pointsset to the determination target image may be projected onto thereference image based on the deformation information, and the number ofpoints among the plurality of points included in the reference image maybe used as the coverage ratio (see FIG. 21).

The deletion determination process based on the coverage ratio may beimplemented by comparing the coverage ratio with a given thresholdvalue. It may be determined that the determination target image can bedeleted when the coverage ratio is equal to or larger than the thresholdvalue, and determined that the determination target image cannot bedeleted when the coverage ratio is smaller than the threshold value. Thedeletion determination process based on the coverage ratio ensures thata certain area (corresponding to the threshold value) captured withinthe determination target image can be covered by the reference imageeven if the determination target image is deleted.

The deletion determination process based on the structural element andthe deletion determination process based on the coverage ratio may becombined in various ways. For example, the deletion determinationprocess based on the structural element and the deletion determinationprocess based on the coverage ratio may be performed independently, andwhether or not the AND condition is satisfied may be determined.Specifically, when it has been determined that the determination targetimage can be deleted based on the structural element, and it has beendetermined that the determination target image can be deleted based onthe coverage ratio, it is determined that the determination target imagecan be deleted. In order to reduce the probability that the attentionarea is missed, it is desirable to delete the determination target imagewhen it has been determined that the determination target image can bedeleted based on both the structural element and the coverage ratioinstead of using the OR condition.

Alternatively, one of the deletion determination process based on thestructural element and the deletion determination process based on thecoverage ratio may be performed first, and the other process may then beperformed. For example, the deletion determination process based on thecoverage ratio is performed first, and the deletion determinationprocess based on the structural element is not performed when it hasbeen determined that the determination target image cannot be deleted.When it has been determined that the determination target image can bedeleted based on the coverage ratio, the deletion determination processbased on the structural element is performed, and whether or not thedetermination target image can be deleted is determined based on theresult of the deletion determination process based on the structuralelement. This makes it possible to reduce the number of images that aresubsequently subjected to the time-series process, and reduce theprocessing load, for example. The processing load can be efficientlyreduced by performing the deletion determination process based on thestructural element or the deletion determination process based on thecoverage ratio that is considered to determine that a larger number ofimages cannot be deleted.

According to the fifth embodiment, the processing section 100 mayperform a first deletion determination process that calculates thecoverage ratio of the determination target image by the reference imagebased on the deformation information about the reference image and thedetermination target image, and determines whether or not thedetermination target image can be deleted based on the calculatedcoverage ratio. The processing section 100 may also perform a seconddeletion determination process that determines whether or not thedetermination target image can be deleted based on the results of theprocess that utilizes the deformation information, and the process thatutilizes the structural element.

According to this configuration, since the deletion determinationprocess can be implemented based on both the first deletiondetermination process and the second deletion determination process, thedetermination accuracy can be improved. In this case, both the firstdeletion determination processes and the second deletion determinationprocess may be performed on each image. Note that the processing loadcan be reduced by performing one of the first deletion determinationprocess and the second deletion determination process first to reducethe number of images subjected to the other of the first deletiondetermination process and the second deletion determination process.

7. Sixth Embodiment

A basic method that utilizes the observation area is described below. Asystem configuration example of an image processing device will bedescribed first, the flow of the process will then be described using aflowchart, and the details of the deletion determination process will bedescribed thereafter using three examples.

7.1 System Configuration Example

FIG. 22 illustrates a system configuration example of the imageprocessing device according to the sixth embodiment. The imageprocessing device includes a processing section 100, an image sequenceacquisition section 200, and a storage section 300.

As illustrated in FIG. 22, the processing section 100 may include areference image selection section 1017, a determination target imageselection section 1018, a deformation information acquisition section1019, an observation area setting section 1020, a corresponding areasetting section 1021, an image feature quantity calculation section1022, a deletion determination section 1023, and a partial imagesequence setting section 1024.

Note that the reference image selection section 1017 and thedetermination target image selection section 1018 are respectively thesame as the reference image selection section 1001 and the determinationtarget image selection section 1002 illustrated in FIG. 1. Thedeformation information acquisition section 1019 acquires thedeformation information about two images.

The observation area setting section 1020 sets part of the determinationtarget image to be the observation area. The observation area may be asquare area having a side of length L, for example. The correspondingarea setting section 1021 deforms the observation area based on thedeformation information acquired by the deformation informationacquisition section 1019 to calculate (set) the corresponding areawithin the reference image.

The image feature quantity calculation section 1022 calculates thefeature quantity of the observation area set by the observation areasetting section 1020, and the feature quantity of the corresponding areaset by the corresponding area setting section 1021. Specific examples ofthe feature quantity are described later.

The deletion determination section 1023 determines whether or not thedetermination target image can be deleted based on the feature quantity(second feature quantity) of the observation area and the featurequantity (first feature quantity) of the corresponding area calculatedby the image feature quantity calculation section 1022. The detailsthereof are described later.

The partial image sequence setting section 1024 sets an image sequencethat is included in the image sequence and includes one or more imagesto be the partial image sequence based on the position of thedetermination target image in the image sequence when the deletiondetermination section 1023 has determined that the determination targetimage cannot be deleted.

7.2 Flow of Process

The flow of the image summarization process according to the sixthembodiment is described below with reference to FIG. 23 (flowchart).When the image summarization process has started, the image sequencethat is subjected to the image summarization process is acquired (S501).

The reference image selection section 1017 selects the first image ofthe input image sequence (i.e., the image sequence acquired in the stepS501, or the partial image sequence set in a step S509) as the referenceimage (S502). The selected reference image is allowed to remain in thesummary image sequence. Note that the process is terminated when thereference image cannot be selected from the input image sequence (e.g.,when no image is included in the image sequence) due to an error or thelike.

The determination target image selection section 1018 selects thedetermination target image from the images included in the input imagesequence (S503). When the determination target image has not been set,the image that immediately follows the reference image (i.e., the secondimage of the input image sequence) is selected as the determinationtarget image. When the kth image of the input image sequence has beenselected as the determination target image, the (k+1)th image (i.e., theselection position is shifted by 1) of the input image sequence isselected as the next determination target image. The process isterminated when the determination target image cannot be selected (e.g.,when the number of images included in the input image sequence is lessthan 2 or k+1).

When the reference image and the determination target image have beenselected, the deformation information acquisition section 1019 acquiresthe deformation information about the reference image and thedetermination target image (S504). The observation area is set withinthe determination target image (S505). When the process in the step S505is performed for the first time after the determination target image hasbeen set in the step S503, the upper left area of the determinationtarget image may be set to be the observation area, for example.

When the observation area has been set, the observation area is deformedbased on the deformation information acquired in the step S504, andprojected onto the reference image to calculate the corresponding area(S506).

The second feature quantity (i.e., the feature quantity of theobservation area) and the first feature quantity (i.e., the featurequantity of the corresponding area) are calculated (S507), and whetheror not the determination target image can be deleted is determined basedon the calculated first feature quantity and the calculated secondfeature quantity (S508). The details of the process in the step S507 andthe process in the step S508 are described later.

When it has been determined that the determination target image can bedeleted in the step S508, the observation area is set again in the stepS505. Specifically, the position of the observation area is updated. Forexample, the position of the observation area within the determinationtarget image may be moved from the upper left area in the rightwarddirection and the downward direction (see FIG. 24). When the observationarea has been set, the steps S506 to S508 are performed again.

When the observation area has reached the lower right end in the stepS505 (i.e., when it has been determined in the step S508 that thedetermination target image can be deleted corresponding to all of theobservation areas set within the determination target image), it isdetermined that the determination target image can be deleted, and thedetermination target image is updated in the step S503.

When it has been determined in the step S508 that the determinationtarget image cannot be deleted while updating the observation area, itis determined that the determination target image cannot be deleted, andthe partial image sequence setting section 1024 sets the partial imagesequence (S509). Specifically, an image sequence that includes thedetermination target image that cannot be deleted, and the subsequentimages is set to be the partial image sequence. When the partial imagesequence has been set, the process in the step S502 is performed usingthe partial image sequence as the input image sequence. The flow of theimage summarization process is the same as that described above withreference to FIGS. 7A to 7D.

Although FIG. 23 illustrates an example in which it is determined thatthe determination target image cannot be deleted when it has beendetermined even once in the step S508 that the determination targetimage cannot be deleted, the configuration is not limited thereto. Forexample, when the observation area is set within the determinationtarget image up to M (M is an integer equal to or larger than 2) times,an integer t that satisfies 1≤t≤M may be set, and it may be determinedthat the determination target image cannot be deleted when it has beendetermined t times or more in the step S508 that the determinationtarget image cannot be deleted. Another method may be used whendetermining the final result from the results of the deletiondetermination process in the step S508 that is performed up to M times.For example, various modifications may be made, such as performing aweighting process that attaches importance to the result obtained usingthe observation area (or the corresponding area) set to the center areaof the image as compared with the result obtained using the observationarea (or the corresponding area) set to the peripheral area of theimage.

7.3 Deletion Determination Process

The details of the feature quantity calculation process in the step S507and the deletion determination process in the step S508 illustrated inFIG. 23 are described below. Note that the observation area and thecorresponding area have been set in advance.

7.3.1 Deletion Determination Based on Brightness Information

A method that utilizes brightness information about the observation areaand the corresponding area as the feature quantity is described below.Various index values may be used as the index value that represents thebrightness information about an area. For example, the RGB values of thepixels within the area may be converted into the brightness values, andthe average value of the brightness values within the area may be usedas the feature quantity. Note that the brightness information about thearea need not necessarily be calculated from the brightness value ofeach pixel by calculating the average value of the brightness values.The median value or the like may also be used. The average value neednot necessarily be calculated by calculating a simple average value. Forexample, a weighted average value may be used, or a trimmed averagevalue that excludes an extreme value may be used.

The RGB value of each pixel may be converted into the brightness valueusing various methods. For example, the maximum value among the R pixelvalue, the G pixel value, and the B pixel value may be used directly asthe brightness value. Note that the brightness value may be calculatedusing another method.

The deletion determination process based on the first feature quantity(i.e., the brightness information about the corresponding area) and thesecond feature quantity (i.e., the brightness information about theobservation area) is described below. Whether or not the determinationtarget image can be deleted is determined based on two conditions.

The first condition is determined by an upper-limit threshold valueK_over and a lower-limit threshold value K_under. Specifically, thefirst feature quantity is compared with the upper-limit threshold valueK_over and the lower-limit threshold value K_under. More specifically,it is determined that the determination target image can be deleted whenthe first feature quantity satisfies the following expression (4), andit is determined that the determination target image cannot be deletedwhen the first feature quantity does not satisfy the expression (4).K_under≤first feature quantity≤K_over  (4)

The lower-limit threshold value K_under may be set so that thebrightness information is smaller than the lower-limit threshold valueK_under when it is difficult to observe the area due to too low abrightness. When the first feature quantity is smaller than thelower-limit threshold value K_under, the corresponding area is normallynot suitable for observation due to blocked up shadows.

The upper-limit threshold value K_over may be set so that the brightnessinformation is larger than the upper-limit threshold value K_over whenit is difficult to observe the area due to too high a brightness. Whenthe first feature quantity is larger than the upper-limit thresholdvalue K_over, the corresponding area is normally not suitable forobservation due to blown out highlights.

Since whether or not blown out highlights or blocked up shadows (or astate close to blown out highlights or blocked up shadows) occur in thecorresponding area can be detected by performing a determination basedon the expression (4), whether or not it is difficult to observe theobject captured within the corresponding area can be determined. Sincethe corresponding area and the observation area are set based on thedeformation information, and have a relationship with the capturedobject area, it is possible to perform appropriate observation byallowing the determination target image to remain in the summary imagesequence, and observing the observation area within the determinationtarget image.

When blown out highlights or blocked up shadows occur in the observationarea within the determination target image, the object cannot beobserved even if the determination target image is allowed to remain inthe summary image sequence. Therefore, it is not advantageous to allowthe determination target image to remain in the summary image sequencein such a case. Therefore, whether or not the following expression (5)is satisfied may be determined in addition to determining whether or notthe expression (4) is satisfied, and it may be determined that thedetermination target image cannot be deleted when the expression (4) isnot satisfied, and the expression (5) is satisfied, otherwise it may bedetermined that the determination target image can be deleted.K_under≤second feature quantity≤K_over  (5)

The second condition is determined based on the difference (the absolutevalue thereof in a narrow sense) between the first feature quantity andthe second feature quantity using a given threshold value K_light.Specifically, it is determined that the determination target imagecannot be deleted when the following expression (6) is satisfied.|First feature quantity−second feature quantity|>K_light  (6)

The absolute value of the difference between the first feature quantityand the second feature quantity is large when the brightness of thecorresponding area and the brightness of the observation area differfrom each other to a large extent (see FIG. 25). It is considered that apreferable brightness when observing the object differs depending on thetype of the object and the like. The object may be easily observed whenthe image is bright, or may be easily observed when the image is dark.The expression (6) is satisfied when one of the first feature quantityand the second feature quantity is large, and the other of the firstfeature quantity and the second feature quantity is small (depending onthe threshold value K_light). This corresponds to an extreme situationin which one of the corresponding area and the observation area isbright, and the other of the corresponding area and the observation areais dark. Specifically, since the brightness of the corresponding area isnot an intermediate value that can be used in a versatile way, it may bedifficult to observe the object captured within the corresponding areadepending on the situation. It is likely that appropriate observationcan be performed when the determination target image that includes theobservation area that significantly differs in brightness from thecorresponding area is allowed to remain in the summary image sequence.

There may be a case where the corresponding area is suitable forobservation, and the observation area is not suitable for observationeven when the expression (6) is satisfied. However, a brightnesssuitable for observation differs depending on the situation, and it isdifficult to determine such a brightness in advance. Therefore, thedetermination target image is allowed to remain in the summary imagesequence when the expression (6) is satisfied. Specifically, the methodaccording to the sixth embodiment allows an unnecessary determinationtarget image to remain in the summary image sequence depending on thesituation.

7.3.2 Deletion Determination Based on Size Information

A method that utilizes size information about the observation area andthe corresponding area as the feature quantity is described below. Thesize information corresponds to the area of each area. For example, thesize information may be calculated by counting the number of pixelsincluded in each area. Note that the size information may be calculatedusing another method.

A situation in which the size of the corresponding area decreases to alarge extent, and the object cannot be appropriately observed (see FIG.26) is taken into account when using the size information as the featurequantity. Therefore, the determination target image is allowed to remainin the summary image sequence when the first feature quantity that isthe size information about the corresponding area is small so that theobject that cannot be appropriately observed within the correspondingarea can be observed within the observation area within thedetermination target image. The deletion determination process may beperformed using only the size of the corresponding area (i.e., usingonly the first feature quantity). In this example, the deletiondetermination process is performed using a relative size (e.g., theratio of the size of the observation area to the size of thecorresponding area). For example, it is determined that the size of thecorresponding area has decreased to a level that is not suitable forobservation, and the determination target image cannot be deleted whenthe following expression (7) is satisfied.(Second feature quantity/first feature quantity)>K_area  (7)

Note that the expression used for the determination is not limited tothe expression (7). It suffices that the expression used for thedetermination be based on the difference between the first featurequantity and the second feature quantity. For example, the differencebetween the logarithm of the second feature quantity and the logarithmof the first feature quantity may be calculated, and compared with agiven threshold value. The difference between the second featurequantity and the first feature quantity may be calculated, and comparedwith a given threshold value.

7.3.3 Deletion Determination Based on Similarity with Given Shape

A method that utilizes the similarity of the observation area and thecorresponding area with a given shape as the feature quantity isdescribed below. The similarity with the given shape represents thedegree by which each area is similar to the given shape. For example,when the given shape is a circle, the degree of circularity calculatedby the following expression (8) may be used as the feature quantity.Note that the given shape is not limited to a circle, and the similaritycalculation method is not limited to the method using the expression(8).Degree of circularity=(4π×area)/(circumferential length)²  (8)

A situation in which the corresponding area has an extreme shape, andthe object cannot be appropriately observed (see FIG. 27) is taken intoaccount when using the similarity with the given shape as the featurequantity. In FIG. 27, the corresponding area has a rectangular shape inwhich the short side is extremely shorter than the long side, and theobject is compressed in the short-side direction. Therefore, a simpleshape (e.g., circle or square) that is suitable for observation is setas the given shape, and the determination target image is allowed toremain in the summary image sequence when the first feature quantitythat is the similarity of the corresponding area with the given shape issmall so that the object that cannot be appropriately observed withinthe corresponding area can be observed within the observation areawithin the determination target image. The deletion determinationprocess may be performed using only the similarity of the correspondingarea (i.e., using only the first feature quantity). In this example, thedeletion determination process is performed using relative information(e.g., the ratio of the similarity of the observation area to thesimilarity of the corresponding area). For example, it is determinedthat the shape of the corresponding area is not suitable forobservation, and the determination target image cannot be deleted whenthe following expression (9) is satisfied.(Second feature quantity/first feature quantity)>K_shape  (9)

Note that the expression used for the determination is not limited tothe expression (9) in the same manner as in the case of using the sizeinformation. For example, the difference between the logarithm of thesecond feature quantity and the logarithm of the first feature quantitymay be calculated, and compared with a given threshold value. Thedifference between the second feature quantity and the first featurequantity may be calculated, and compared with a given threshold value.

7.3.4 Combination of Determinations that Utilize a Plurality of FeatureQuantities

Two or more deletion determination processes among the deletiondetermination process that utilizes the brightness information, thedeletion determination process that utilizes the size information, andthe deletion determination process that utilizes the similarity with agiven shape may be used in combination.

The deletion determination processes may be combined in various ways. Ifa priority is given to prevention of occurrence of an area that is notsuitable for observation, the probability that it is determined that thedetermination target image cannot be deleted is increased. Therefore,when using a plurality of types of feature quantities, the deletiondetermination process is performed using each feature quantity, and thedetermination target image is deleted when it has been determined byeach deletion determination process that the determination target imagecan be deleted. The determination target image is not deleted when ithas been determined by at least one deletion determination process thatthe determination target image cannot be deleted. According to thisconfiguration, the determination target image is allowed to remain whenit has been determined from at least one feature quantity that thecorresponding area is not suitable for observation, and it is likelythat the target object can be appropriately observed.

However, when it is easily determined that the determination targetimage cannot be deleted, the number of images that are allowed to remainin the summary image sequence increases, and the effect of reducing thenumber of images may decrease. Therefore, when a priority is given to areduction in the number of images, a method may be used that determinesthat the determination target image can be deleted with higherprobability.

For example, even when the determination target image is not deletedwhen it has been determined by at least one deletion determinationprocess among a plurality of deletion determination processes that thedetermination target image cannot be deleted, the determinationcondition used for the deletion determination process that utilizes eachfeature quantity may be changed. For example, when using the sizeinformation and the similarity with a given shape in combination, thevalue K_area and the value K_shape may be increased as compared with thecase of independently using the size information and the similarity witha given shape. In this case, since it is easily determined by eachdeletion determination process that the determination target image canbe deleted, it is possible to prevent a situation in which the number ofimages included in the summary image sequence increases to a largeextent while implementing a determination based on the size and thesimilarity.

According to the sixth embodiment, the image processing device includesthe image sequence acquisition section 200 that acquires an imagesequence that includes a plurality of images, and the processing section100 that performs the image summarization process that deletes some ofthe plurality of images included in the image sequence acquired by theimage sequence acquisition section 200 to acquire a summary imagesequence (see FIG. 22). The processing section 100 selects the referenceimage and the determination target image used for the imagesummarization process from the plurality of images, sets the observationarea within the determination target image, and calculates thecorresponding area (i.e., an area of the reference image thatcorresponds to the observation area) based on the deformationinformation about the reference image and the determination targetimage. The processing section 100 determines whether or not thedetermination target image can be deleted based on at least one of thefirst feature quantity calculated from the corresponding area and thesecond feature quantity calculated from the observation area.

The observation area is an area that is set within the determinationtarget image. The observation area is an area narrower than thedetermination target image in a narrow sense. When determining whetheror not the entirety of the object captured within the determinationtarget image is suitable for observation within the reference image, itis necessary to set the observation area a plurality of times to coverthe entire determination target image while changing the position of theobservation area within the determination target image when thereference image and the determination target image have been selected.For example, each pixel included in the determination target image isincluded in the observation area at least once. This is implemented bysetting the observation area while shifting the observation area by onepixel in the rightward direction and the downward direction from theupper left area (see FIG. 24). However, since the processing loadincreases as the observation area is set a larger number of times, theamount of calculations may be reduced by shifting the observation areaby the length of one side of the observation area.

When using the size information as the feature quantity, an area that isnecessary and sufficient for appropriate observation may be set to bethe observation area. In this case, since the second feature quantityrepresents an appropriate reference value, whether or not thecorresponding area has a size suitable for observation can be determinedby comparing the second feature quantity with the first featurequantity, for example. When using the similarity with a given shape(i.e., a shape (e.g., circle or square) that is suitable forobservation) as the feature quantity, the observation area may have ashape similar to the given shape (the same shape as the given shape in anarrow sense). It is considered that an area having a constant size anda constant shape is used as the observation area during a series ofprocesses. Note that the observation area may be set variably.

This makes it possible to determine whether or not the determinationtarget image can be deleted based on the feature quantity of theobservation area within the determination target image, and the featurequantity of the corresponding area within the reference image. When thecorresponding area is an area calculated by deforming the observationarea based on the deformation information, the object captured withinthe observation area corresponds to the object captured within thecorresponding area. Therefore, when it has been determined that thecorresponding area is not suitable for observation based on the featurequantity, it suffices to allow the observation area in which thecorresponding object (identical object in a narrow sense) is captured tobe observed after the image summarization process. This is implementedby allowing the determination target image to remain in the summaryimage sequence (i.e., determining that the determination target imagecannot be deleted).

The first feature quantity may be at least one of the brightnessinformation about the corresponding area, the size information about thecorresponding area, and the similarity information about the similarityof the corresponding area with a given (specific) shape. The secondfeature quantity may be at least one of the brightness information aboutthe observation area, the size information about the observation area,and the similarity information about the similarity of the observationarea with a given shape.

The similarity of a given area with a given shape is an index value thatrepresents the degree by which the given area is similar to the givenshape. For example, when the given shape is a circle, a k-sided regularpolygonal area (k is an integer equal to or larger than 3) has a lowerdegree of similarity with the given shape as the value k decreases, andhas a higher degree of similarity with the given shape as the value kincreases. When the given shape is symmetrical (e.g., circle), asymmetrical area tends to have a higher degree of similarity with thegiven shape as compared with an asymmetrical area.

This makes it possible to use at least one of the brightnessinformation, the size information, and the similarity information aboutthe similarity with a given shape as the feature quantity. Specifically,the image processing device according to the sixth embodiment determineswhether or not the object within the corresponding area is suitable forobservation based on the brightness, the size, or the shape, and allowsthe determination target image to remain in the summary image sequencewhen the object within the corresponding area is not suitable forobservation. It may be determined that the object within thecorresponding area is not suitable for observation when the brightnessis too high (blown out highlights), when the brightness is too low(blocked up shadows), when the size is extremely small, or when theshape is extreme (e.g., when the shape is deformed or distorted), forexample.

The processing section 100 may determine whether or not thedetermination target image can be deleted based on at least onecomparison process among a first comparison process that compares thefirst feature quantity with a first threshold value, a second comparisonprocess that compares the second feature quantity with a secondthreshold value, and a third comparison process that compares the degreeof difference between the first feature quantity and the second featurequantity with a third threshold value.

The degree of difference between the first feature quantity and thesecond feature quantity that represents the difference between the firstfeature quantity and the second feature quantity is calculated by thedifference, the ratio, or a value corresponding thereto (e.g., thedifference between the logarithm of the first feature quantity and thelogarithm of the second feature quantity). When using the brightnessinformation as the feature quantity, the degree of difference betweenthe first feature quantity and the second feature quantity is high whenone of the corresponding area and the observation area is bright, andthe other of the corresponding area and the observation area is dark.When using the size information as the feature quantity, the degree ofdifference between the first feature quantity and the second featurequantity is high when one of the corresponding area and the observationarea has a large area, and the other of the corresponding area and theobservation area has a small area. When using the similarity with thegiven shape as the feature quantity, the degree of difference betweenthe first feature quantity and the second feature quantity is high whenone of the corresponding area and the observation area has a shapesimilar to the given shape, and the other of the corresponding area andthe observation area has a shape that differs to a large extent from thegiven shape.

This makes it possible to perform the deletion determination processbased on at least one comparison process among the first to thirdcomparison processes. The first comparison process is performed based onthe first feature quantity that is the feature quantity of thecorresponding area. For example, the first comparison process may beused when a determination can be made independently of the state of theobservation area (e.g., when determining whether or not blown outhighlights or blocked up shadows occur in the corresponding area (seethe expression (4)). The second comparison process is performed based onthe second feature quantity that is the feature quantity of theobservation area. For example, the second comparison process may be usedwhen determining whether or not blown out highlights or blocked upshadows occur in the observation area (see the expression (5)). Thesecond comparison process may be used alone. However, it is desirable touse the second comparison process in combination with the firstcomparison process or the like taking account of the effect of reducingthe number of images through the image summarization process. Since thethird comparison process is based on the degree of difference betweenthe first feature quantity and the second feature quantity, the thirdcomparison process takes account of the state of the corresponding areaand the state of the observation area. It is possible to perform anaccurate deletion determination process by utilizing the thirdcomparison process. Note that the two feature quantities used for thethird comparison process must correspond to each other (e.g., acomparison between the brightness information about the correspondingarea and the size information about the observation area is not useful).Specifically, when one of the two feature quantities used for the thirdcomparison process is the brightness information, the other of the twofeature quantities used for the third comparison process is also thebrightness information. When one of the two feature quantities used forthe third comparison process is the size information, the other of thetwo feature quantities used for the third comparison process is also thesize information. When one of the two feature quantities used for thethird comparison process is the similarity with the given shape, theother of the two feature quantities used for the third comparisonprocess is also the similarity with the given shape.

The processing section 100 may calculate the brightness informationabout the corresponding area as the first feature quantity based on thepixel value of each pixel included in the corresponding area, andcalculate the brightness information about the observation area as thesecond feature quantity based on the pixel value of each pixel includedin the observation area.

This makes it possible to perform the deletion determination processusing the brightness information as the feature quantity. The brightnessinformation about the corresponding area and the brightness informationabout the observation area are calculated based on the brightnesscalculated corresponding to each pixel included in each area. Forexample, the maximum value among the R pixel value, the G pixel value,and the B pixel value is used as the brightness of each pixel. Note thatthe brightness of each pixel may be calculated using another method(e.g., a method that calculates the average value of the maximum valueand the minimum value). The brightness information about each area maybe calculated from the brightness of each pixel by calculating theaverage value of the brightness of each pixel included in each area.Note that the brightness information about each area may be calculatedusing another method (e.g., a method that calculates the median value, aweighted average value, or a trimmed average value). Specific examplesof the deletion determination process based on the brightnessinformation include a process that determines that the determinationtarget image cannot be deleted when the corresponding area is notsuitable for observation due to blown out highlights or blocked upshadows. In this case, the determination target image that includes theobservation area in which the same object as that captured within thecorresponding area is considered to be captured can be allowed to remainin the summary image sequence, and the object can be appropriatelyobserved.

The processing section 100 may determine that the determination targetimage cannot be deleted when the first feature quantity that is thebrightness information is larger than a given upper-limit thresholdvalue, or when the first feature quantity is smaller than a givenlower-limit threshold value.

This makes it possible to implement the process represented by theexpression (4). This process takes account of a situation in which blownout highlights or blocked up shadows (or a state close to blown outhighlights or blocked up shadows) occur in the corresponding area.Therefore, the upper-limit threshold value may be set so that thebrightness information is larger than the upper-limit threshold valuewhen the object is not suitable for observation due to too high abrightness, and the lower-limit threshold value may be set so that thebrightness information is smaller than the lower-limit threshold valuewhen the object is not suitable for observation due to too low abrightness.

The processing section 100 may calculate a value represented by the sizeinformation about the corresponding area as the first feature quantity,and calculate a value represented by the size information about theobservation area as the second feature quantity.

This makes it possible to perform the deletion determination processusing the size information as the feature quantity. For example,information that corresponds to the area of each area may be used as thesize information. Specifically, the size information may be calculatedby counting the number of pixels included in each area. Specificexamples of the deletion determination process based on the sizeinformation include a process that determines that the determinationtarget image cannot be deleted when the corresponding area has a verysmall area, and is not suitable for observation (i.e., the object cannotbe appropriately observed) (see FIG. 26).

The processing section 100 may calculate a value that represents thesimilarity of the corresponding area with a given shape as the firstfeature quantity, and calculate a value that represents the similarityof the observation area with a given shape as the second featurequantity.

This makes it possible to perform the deletion determination processusing the similarity with the given shape as the feature quantity. Whenthe given shape is a circle, the degree of circularity represented bythe expression (8) may be used as the similarity with the given shape,for example. Specific examples of the deletion determination processbased on the similarity with the given shape include a process thatdetermines that the determination target image cannot be deleted whenthe corresponding area has an extreme shape, and is not suitable forobservation (i.e., the object is deformed in the vertical direction)(see FIG. 27).

The processing section 100 may determine that the determination targetimage cannot be deleted when the degree of difference between the firstfeature quantity and the second feature quantity is larger than a giventhreshold value.

This makes it possible to perform the deletion determination processusing the feature quantity of the corresponding area and the featurequantity of the observation area. When using the brightness informationas the feature quantity, the degree of difference between the firstfeature quantity and the second feature quantity is high when one of thecorresponding area and the observation area is bright, and the other ofthe corresponding area and the observation area is dark. The brightnessvalue suitable for observation cannot be uniquely determined. Forexample, the brightness value suitable for observation varies dependingon the relationship with the shape/color of the object, the color of thebackground that is captured behind the object, or the like. If thebrightness information about the corresponding area represents anintermediate value, the brightness information may be used in aversatile ways to a certain extent. However, it is not considered thatthe brightness information can be used in a versatile ways when thedegree of difference is high. Specifically, when the degree ofdifference is high (e.g., when the expression (6) is satisfied), it isnecessary to consider that the corresponding area may have a brightnessthat is not suitable for observation, and it is desirable to determinethat the determination target image cannot be deleted. Note that thedetermination target image that includes the observation area that isnot suitable for observation may be allowed to remain in the summaryimage sequence due to the process that utilizes the expression (6) orthe like even when the corresponding area is suitable for observation.

When using the size information as the feature quantity, whether or notthe corresponding area has a small area is taken into consideration, andit is not effective to allow the determination target image to remain inthe summary image sequence when the observation area does not have anarea suitable for observation. Therefore, it is desirable to detect thatthe corresponding area is sufficiently smaller than the observation area(see the expression (7)) as a situation in which the degree ofdifference is high. When using the size information as the featurequantity, it is considered that the second feature quantity is constantwhen the size and the shape of the observation area are constant, and avalue calculated in advance may be continuously used. In such a case,since the second feature quantity in the expression (7) is constant, thedetermination process is substantially performed based on the firstfeature quantity. However, when the observation area is set dynamically(e.g., when the observation area changes according to an instructionissued by the user, or an area corresponding to the attention areadetected by image processing is set to be the observation area), it isnecessary to calculate the second feature quantity each time theobservation area is set.

When using the similarity with a given shape (that is suitable forobservation) as the feature quantity, whether or not the correspondingarea has an extreme shape is taken into consideration, and it is noteffective to allow the determination target image to remain in thesummary image sequence when the shape of the observation area is notclose to the given shape to a certain extent. Therefore, it is desirableto detect that the similarity of the corresponding area is sufficientlysmaller than the similarity of the observation area (see the expression(9)) as a situation in which the degree of difference is high. In thiscase, the second feature quantity can be calculated in advance when theshape of the observation area is constant. For example, an area havingthe same shape as the given shape may be set to be the observation area.

8. Seventh Embodiment

A method that utilizes the deletion determination process that utilizesthe first feature quantity and the second feature quantity (describedabove in connection with the sixth embodiment) (hereinafter referred toas “first deletion determination process”) in combination with a seconddeletion determination process that differs from the first deletiondetermination process is described below.

Specifically, the second deletion determination process is performed oneach image included in the image sequence acquired by the image sequenceacquisition section 200 to set a summary candidate image sequence thatincludes summary images that have been determined to be allowed toremain, and a deletion candidate image sequence that includes deletioncandidate images that have been determined to be deleted. The firstdeletion determination process is then performed to generate the summaryimage sequence. When performing the first deletion determinationprocess, the reference image is selected from the summary candidateimage sequence, and the determination target image is selected from thedeletion candidate image sequence.

Specifically, the method according to the seventh embodiment implementsa two-step process that performs the second deletion determinationprocess as preprocessing to provisionally determine the images that areallowed to remain and the images that are deleted, and performs thefirst deletion determination process using the provisional results todetermine the final results. This makes it possible to improve thedetermination accuracy as compared with the case of performing the imagesummarization process based on only one of the first deletiondetermination processes and the second deletion determination process,for example. Specifically, since the second deletion determinationprocess utilizes a process that differs from the first deletiondetermination process, the process can be implemented from a differentpoint of view.

For example, the determination process based on the structural element(or both the determination process based on the structural element andthe determination process based on the coverage ratio, as required) maybe performed as the second deletion determination process. An imagesummarization process that suppresses a situation in which the attentionarea is missed can be implemented by the determination process based onthe structural element. In this case, however, an area that is difficultto observe may occur (see FIG. 26). Specifically, an image that shouldnot be deleted may be deleted by the second deletion determinationprocess that utilizes the structural element. However, it is possible torecover such an image by performing the first deletion determinationprocess using the results of the second deletion determination process,and implement a more appropriate image summarization process.

A system configuration example of an image processing device will bedescribed first, and the flow of the process will then be describedusing a flowchart. Note that the second deletion determination processis implemented using a method among the methods described above inconnection with the first to fifth embodiments, and detailed descriptionthereof is omitted.

8.1 System Configuration Example

FIG. 28 illustrates a system configuration example of the imageprocessing device according to the seventh embodiment. In FIG. 28, thepartial image sequence setting section 1024 (see FIG. 22) is omittedfrom the processing section 100, and the processing section 100 furtherincludes a summary candidate image sequence generation section 1016.Note that detailed description of the same configuration as thatdescribed above in connection with the sixth embodiment is omitted.

The summary candidate image sequence generation section 1016 performsthe second deletion determination process on the image sequence acquiredby the image sequence acquisition section 200 to generate the summarycandidate image sequence that includes the summary images (i.e., imagesthat are allowed to remain in the summary image sequence). The summarycandidate image sequence generation section 1016 may set the deletioncandidate image sequence that includes the deletion candidate images(i.e., images that are included in the image sequence acquired by theimage sequence acquisition section 200, and are not allowed to remain inthe summary candidate image sequence).

The determination target image selection section 1018 according to theseventh embodiment selects the determination target image from theimages included in the deletion candidate image sequence. The referenceimage selection section 1017 selects the reference image from the imagesincluded in the summary candidate image sequence corresponding to theposition of the determination target image selected by the determinationtarget image selection section 1018 within the image sequence (i.e., theimage sequence acquired by the image sequence acquisition section 200).The details of the process performed by the reference image selectionsection 1017 and the process performed by the determination target imageselection section 1018 are described later.

As described above, the image that is allowed to remain in the summaryimage sequence (or a candidate for the image that is allowed to remainin the summary image sequence) is selected as the reference image, andthe image that is subjected to the deletion determination process basedon the reference image is selected as the determination target image.Specifically, when using the coverage ratio during the second deletiondetermination process, the reference image and the determination targetimage are set during the second deletion determination process, and thereference image and the determination target image are also set duringthe first deletion determination process (i.e., the process thatutilizes the observation area). Note that the reference image and thedetermination target image that are set during the first deletiondetermination process may respectively be referred to as “first image”and “second image” when it is unclear whether the reference image andthe determination target image are set during the first deletiondetermination process or the second deletion determination process.

8.2 Flow of Process

The flow of the image summarization process according to the seventhembodiment is described below with reference to FIG. 29 (flowchart).When the image summarization process has started, the image sequencethat is subjected to the image summarization process is acquired (S601).The image sequence acquired in the step S601 may be referred to as“acquired image sequence” in order to clearly distinguish the imagesequence acquired in the step S601 from another image sequence.

The second deletion determination process is performed on the acquiredimage sequence to set the summary candidate image sequence and thedeletion candidate image sequence (S602). A specific example of thesecond deletion determination process performed in the step S602 isdescribed later. For example, when it has been determined that the thirdand eighth images among twelve images included in the acquired imagesequence cannot be deleted, and the remaining images can be deleted (seeFIG. 30), the summary candidate image sequence includes the third andeighth images, and the deletion candidate image sequence includes theremaining ten images. The deletion determination process may beperformed a plurality of times on a single image depending on the seconddeletion determination process. Note that the deletion determinationresults refer to the final results when the second deletiondetermination process has completed, and it is not necessarilyimmediately determined that an image is the deletion candidate image orthe summary image when it has been determined that the image can bedeleted or cannot be deleted only once.

After completion of the second deletion determination process, thedetermination target image is selected (S603). For example, thedetermination target image is sequentially selected from the first imageof the deletion candidate image sequence. Therefore, the first image ofthe deletion candidate image sequence is selected when the process inthe step S603 is performed for the first time. The determination targetimage is updated (i.e., the image included in the deletion candidateimage sequence that immediately follows the current determination targetimage is selected) during the subsequent process in the step S603.

When the determination target image has been selected, the referenceimage is selected from the summary candidate image sequencecorresponding to the position of the determination target image withinthe acquired image sequence (S604). For example, the summary imageincluded in the acquired image sequence that precedes and is situatedclosest to the determination target image is selected as the firstimage, and the summary image included in the acquired image sequencethat follows and is situated closest to the determination target image,is selected as the second image. When no summary image precedes orfollows the determination target image, the corresponding referenceimage is not selected.

In the example illustrated in FIG. 30, when the first image of thedeletion candidate image sequence has been selected as the determinationtarget image, the first reference image is not selected since no summaryimage precedes the determination target image in the acquired imagesequence. The third image (i.e., the first image of the summarycandidate image sequence) among the third and eighth images (summaryimages) of the acquired image sequence that follows the determinationtarget image and is situated closest to the determination target imageis selected as the second image.

In the example illustrated in FIG. 30, when the third to sixth images ofthe deletion candidate image sequence are selected as the determinationtarget image, the first image of the summary candidate image sequence isselected as the first reference image, and the second image of thesummary candidate image sequence is selected as the second referenceimage. When the seventh to tenth images of the deletion candidate imagesequence are selected as the determination target image, the secondimage of the summary candidate image sequence is selected as the firstreference image, and the second reference image is not selected.

The deformation information acquisition process (S605), the observationarea selection process (S606), the corresponding area selection process(S607), the image feature quantity calculation process (S608), and thedeletion determination process (S609) after the reference image and thedetermination target image have been selected are performed in the samemanner as in the steps S504 to S508 illustrated in FIG. 23,respectively.

When two reference images have been selected, the first deletiondetermination process using the first reference image and thedetermination target image, and the first deletion determination processusing the second reference image and the determination target image areperformed, and it is determined that the determination target imagecannot be deleted (i.e., is allowed to remain in the summary imagesequence) when it has been determined by each first deletiondetermination process that the determination target image cannot bedeleted. Specifically, even when the corresponding area within onereference image is not suitable for observation, it is not advantageousto allow the determination target image to remain in the summary imagesequence when the corresponding area within the other reference image inwhich the same object is captured is suitable for observation.

In the seventh embodiment, since the images that are allowed to remainhave been provisionally searched during the second deletiondetermination process, the partial image sequence setting process (S509in FIG. 23) (see the sixth embodiment) is unnecessary.

According to the seventh embodiment, the processing section 100 sets thesummary candidate image sequence that includes the summary images amongthe plurality of images included in the image sequence acquired by theimage sequence acquisition section 200 that have been determined to beallowed to remain based on the results of the process that utilizes thedeformation information and the process that utilizes the structuralelement that corresponds to the attention area, and sets the deletioncandidate image sequence that includes the deletion candidate imagesthat have been determined to be deleted based on the results of theprocess that utilizes the deformation information and the process thatutilizes the structural element that corresponds to the attention area.The processing section 100 selects the first image from the summarycandidate image sequence, selects the second image from the deletioncandidate image sequence, sets the observation area within the secondimage, calculates the corresponding area that is an area within thefirst image that corresponds to the observation area based on thedeformation information about the first image and the second image, anddetermines whether or not the second image can be deleted based on atleast one of the first feature quantity calculated from thecorresponding area and the second feature quantity calculated from theobservation area.

The first image is the reference image that is set during the firstdeletion determination process, and the second image is thedetermination target image that is set during the first deletiondetermination process.

This makes it possible to perform the process that sets the summarycandidate image sequence and the deletion candidate image sequence fromthe image sequence (acquired image sequence) (see FIG. 30) aspreprocessing for the deletion determination process (first deletiondetermination process) that utilizes the first feature quantity and thesecond feature quantity. It is determined that the summary image isallowed to remain in the summary image sequence by selecting thereference image from the summary candidate image sequence, and selectingthe determination target image from the deletion candidate imagesequence, and it is possible to finally determine whether to delete thedeletion candidate image or allow the deletion candidate image to remainin the summary candidate image sequence. Therefore, the accuracy of theprocess that sets the summary candidate image sequence and the deletioncandidate image sequence may be low to some extent. For example, eachimage among the plurality of images included in the acquired imagesequence that cannot be clearly determined to be the summary image maybe determined to be the deletion candidate image. Since a high-accuracydetermination is then made by performing the first deletiondetermination process, each deletion candidate image may be allowed toremain in the summary image sequence. Since it is considered that theimage that has been determined to be the summary image is allowed toremain in the summary image sequence, it is desirable that the processthat sets the summary candidate image sequence and the deletioncandidate image sequence be performed based on a standard (instead ofbeing performed randomly) taking account of the final results of theimage summarization process (e.g., the effect of reducing the number ofimages) and the like.

9. Eighth Embodiment

A method that sets the partial image sequence using a scene change isdescribed below.

9.1 System Configuration Example

FIG. 31 illustrates a system configuration example of an imageprocessing device according to the eighth embodiment. The imageprocessing device includes a processing section 100, an image sequenceacquisition section 200, and a storage section 300.

As illustrated in FIG. 31, the processing section 100 may include adeformation information acquisition section 1025, a scene changedetection section 1026, a partial image sequence setting section 1027, areference image selection section 1028, a determination target imageselection section 1029, and a deletion determination section 1030.

The deformation information acquisition section 1025 acquires thedeformation information about two images. The scene change detectionsection 1026 detects a scene change from the acquired image sequence. Aspecific method is described later.

The partial image sequence setting section 1027 sets part of the imagesequence to be the partial image sequence based on the scene changedetected by the scene change detection section 1026. Specifically, theposition of the scene change may be used as the starting point or theend point of the partial image sequence. For example, when three scenechanges A1 to A3 have been detected form the image sequence (see FIG.32A), a partial image sequence B1 from the first image of the imagesequence to the scene change A1, a partial image sequence B2 from thescene change A1 to the scene change A2, a partial image sequence B3 fromthe scene change A2 to the scene change A3, and a partial image sequenceB4 from the scene change A3 to the final image of the image sequence,may be set. More specifically, when each scene change is set betweenadjacent images (see FIG. 32B), the starting point and the end point ofeach partial image sequence correspond to the image that immediatelyprecedes or follows the scene change. In the example illustrated in FIG.32B, the partial image sequence B1 corresponds to the series of imagesfrom the first image of the image sequence to the image that immediatelyprecedes the scene change A1, and the partial image sequence B2corresponds to the series of images from the image that immediatelyfollows the scene change A1 to the image that immediately precedes thescene change A2. When a plurality of partial image sequences are set,the process performed by each section is respectively performed on eachpartial image sequence.

The reference image selection section 1028 selects the reference imagefrom a plurality of images included in the partial image sequence. Thedetermination target image selection section 1029 selects an image amongthe plurality of images included in the partial image sequence thatdiffers from the reference image as the determination target image.

The deletion determination section 1030 determines whether or not thedetermination target image can be deleted based on the deformationinformation about the reference image and the determination targetimage. In the eighth embodiment, whether or not the determination targetimage can be deleted is determined based on the structural element.

As illustrated in FIG. 33, the deletion determination section 1030 mayinclude a structural element generation section 1034, a coverage areacalculation section 1031, and an attention area miss probabilitydetermination section 1035. Note that the configuration of the deletiondetermination section 1030 is not limited to the configurationillustrated in FIG. 33. Various modifications may be made, such asomitting some of the elements illustrated in FIG. 33, or adding otherelements.

The structural element generation section 1034 generates the structuralelement used for the process performed by the attention area missprobability determination section 1035 based on the attention area. Forexample, an area having the same shape and the same size as those of theattention area is set to be the structural element. Note that thestructural element is not limited thereto.

The coverage area calculation section 1031 calculates the coverage area,and may set an area of the determination target image other than thecoverage area to be the non-coverage area.

The attention area miss probability determination section 1035 performsa determination process that determines the probability that theattention area captured within the determination target image is notcaptured within the reference image (i.e., the attention area is missed)when the determination target image is deleted.

9.2 Flow of Process

The flow of the image summarization process according to the eighthembodiment is described below with reference to FIG. 34 (flowchart).When the image summarization process has started, the image sequenceacquisition section 200 acquires the image sequence that is subjected tothe image summarization process (S701).

A scene change is detected from the acquired image sequence (S702).Specifically, whether or not each image included in the image sequenceis suitable for calculation of the deformation information isdetermined, and it is determined that a scene change occurred at aposition corresponding to an image that has been determined not to besuitable for calculation of the deformation information. For example,when an image selected from the image sequence is not suitable forcalculation of the deformation information (see FIG. 35A), it may bedetermined that a scene change occurred between the selected image andthe image that immediately follows the selected image. In this case, theselected image is the end point of the partial image sequence, and theimage that immediately follows the selected image is the starting pointof the next partial image sequence. It may be determined that a scenechange occurred at positions that precede or follow the selected image.

Whether or not the selected image is suitable for calculation of thedeformation information may be determined by determining whether or noteach pixel of the selected image is suitable for calculation of thedeformation information, and calculating the ratio of the pixels thathave been determined to not be suitable for calculation of thedeformation information to the pixels that have been determined to besuitable for calculation of the deformation information. For example,when a given area that includes the processing target pixel(determination target pixel) has a small amount of texture, it isdifficult to distinguish the processing target pixel and its peripheralpixel (see FIG. 36). Therefore, even if the selected image has beenselected as the reference image, and an area having similar propertieshas been found within the determination target image, it is difficult todetermine the pixel of the determination target image that correspondsto the processing target pixel, and calculate accurate deformationinformation. For example, texture information (e.g., edge quantity)about the given area that includes the processing target pixel iscalculated, and it is determined that the processing target pixel issuitable for calculation of the deformation information when a valuerepresented by the texture information is larger than a given thresholdvalue.

Each image included in the image sequence may be sequentially selectedas the selected image, the ratio of the number of pixels suitable forcalculation of the deformation information to the total number of pixelsmay be calculated for the selected image, it may be determined that theselected image is suitable for calculation of the deformationinformation (i.e., no scene change occurred at a position correspondingto the selected image) when the calculated ratio is larger than a giventhreshold value, and it may be determined that the selected image is notsuitable for calculation of the deformation information (i.e., a scenechange occurred at a position corresponding to the selected image) whenthe calculated ratio is equal to or smaller than the given thresholdvalue.

When a scene change has been detected, the partial image sequence is setbased on the detected scene change (S703). As illustrated in FIGS. 32Aand 32B, the starting point and the end point of the partial imagesequence may be set corresponding to the position of the detected scenechange.

In the step S703, the images included in the set partial image sequencethat have not been subjected to the image summarization process areselected. The reference image selection section 1028 selects the firstimage of the selected partial image sequence as the reference image(S704). When the subsequent process in the step S704 is performed on agiven partial image sequence (i.e., when the process in the step S704 isperformed after the process in the step S707), the determination targetimage that has been determined to be allowed to remain in the deletiondetermination process in the step S707 is selected as the next referenceimage. The selected reference image is allowed to remain in the summaryimage sequence. Note that the process on the partial image sequence isterminated when the reference image cannot be selected from the partialimage sequence due to an error or the like.

When the reference image has been selected, the determination targetimage selection section 1029 selects the determination target image fromthe images included in the partial image sequence (S705). When thedetermination target image has not been set, the image that immediatelyfollows the reference image (i.e., the second image of the partial imagesequence when the process in the step S705 is performed on a givenpartial image sequence for the first time) is selected as thedetermination target image. When the kth image of the partial imagesequence has been selected as the determination target image, the(k+1)th image (i.e., the selection position is shifted by 1) of theinput image sequence is selected as the next determination target image.When the deletion determination process has been performed on the lastimage of the partial image sequence, the determination target imagecannot be selected in the step S705. In this case, the imagesummarization process on the partial image sequence is terminated, andthe step S703 is performed again.

When the reference image and the determination target image have beenselected, the deformation information about the reference image and thedetermination target image is calculated (S706), and whether or not thedetermination target image can be deleted is determined based on thecalculated deformation information (S707). Whether or not thedetermination target image can be deleted may be determined using onemethod or two or more methods among the methods described above inconnection with the first to fifth embodiments.

When it has been determined that the determination target image can bedeleted in the step S707, the determination target image is updated(S705). When it has been determined that the determination target imagecannot be deleted (i.e., the determination target image cannot becovered by the reference image) in the step S707, it is necessary toallow the determination target image to remain in the summary imagesequence. Therefore, the determination target image that has beendetermined to be allowed to remain in the step S707 is selected as thenext reference image in the step S704.

The image summarization process on one partial image sequence iscompleted by the processes in the steps S704 to S707. In this case, theimage summarization process is performed on the next partial imagesequence from the step S703. When the image summarization process hasbeen performed on each partial image sequence (i.e., when the partialimage sequence cannot be selected in the step S703), the process isterminated.

The flow of the image summarization process on each partial imagesequence is the same as that described above with reference to FIGS. 7Ato 7D.

Although FIG. 34 illustrates an example in which a plurality of partialimage sequences that have been set based on a scene change aresequentially processed one by one, the configuration is not limitedthereto. When the configuration of the processing section 100 issuitable for parallel processing (e.g., when a CPU that includes aplurality of cores is used as the processing section 100), or when theimage processing device according to the eighth embodiment includes aplurality of computers, and distributed processing is performed by eachcomputer, the deletion determination process (S704 to S707) may beperformed on the plurality of partial image sequences in parallel. Thismakes it possible to reduce the time required for the deletiondetermination process, for example.

9.3 Modifications

Various modifications may be made of the method according to the eighthembodiment. For example, the scene change detection method is notlimited to the above method. For example, a scene change may be detectedbased on similarity information about two images (two adjacent images ina narrow sense) included in the image sequence acquired by the imagesequence acquisition section 200. Since it is considered that thedeformation information can be accurately estimated when the similaritybetween the images is high, the similarity information about the imagescan be used as information that represents the accuracy of thedeformation information. In this case, NCC, or the reciprocal of SSD orSAD (degree of difference between images) may be used as the similarityinformation about the images. It is determined that a scene changeoccurred between two images when a value represented by the similarityinformation is smaller than a given threshold value. For example, whenit has been determined that the similarity between a first image and asecond image that is adjacent to the first image is low, it may bedetermined that a scene change occurred between the first image and thesecond image (see FIG. 39). It is desirable that two images for whichthe similarity information is calculated be adjacent to each other inthe image sequence in order to uniquely determine the position of ascene change.

Note that the reference image/determination target image selectionmethod employed in the steps S704 and S705 in FIG. 34 is not limited tothe above method.

According to the eighth embodiment, the image processing device includesthe image sequence acquisition section 200 that acquires an imagesequence that includes a plurality of images, and the processing section100 that performs the image summarization process that deletes some ofthe plurality of images included in the image sequence acquired by theimage sequence acquisition section 200 to acquire a summary imagesequence (see FIG. 31). The processing section 100 detects a scenechange from the image sequence, and sets the partial image sequence thatincludes images among the plurality of images included in the imagesequence based on the detected scene change. The processing section 100selects the reference image and the determination target image from thepartial image sequence, and determines whether or not the determinationtarget image can be deleted based on the deformation information aboutthe reference image and the determination target image.

The term “scene change” refers to a change in scene corresponding to thecaptured image. A scene change is widely used in the field of a moviedivision technique (e.g., insertion of chapter information) and thelike. A scene change used in connection with a known method may be useddirectly as the scene change according to the eighth embodiment. In thiscase, the processing section 100 may detect a scene change based on oneof motion information calculated from a plurality of images, imaginginformation about a specific object, and brightness information.

The motion information represents a change in position of the objectbetween two images (between two adjacent images in a narrow sense). Forexample, the motion information may be the motion vector. The motionvector may be calculated using various methods. The motion vector may besimply calculated by performing a block matching process on a given area(block) within one image and the other image. Specifically, informationthat represents the relative positional relationship between theposition of the given area within the image and the position of thematched area within the image is the motion vector. When using themotion vector, it may be determined that a scene change occurred betweentwo images used to calculate the motion vector when the motion vector islarge (e.g., the motion vector is compared with a given thresholdvalue).

The imaging information about a specific object is information thatrepresents whether or not a characteristic object is captured. Theobject may be detected from the image using various methods. Forexample, information about the target specific object may be stored as atemplate, and a template matching process may be performed on eachimage. In this case, a change from a state in which the specific objectis captured to a state in which the specific object is not captured, ora change from a state in which the specific object is not captured to astate in which the specific object is captured, is detected as a scenechange.

A scene change may be detected using the brightness information. Forexample, when using an RGB channel image, the maximum value among the Rvalue, the G value, and the B value of each pixel may be calculated asthe brightness value, and the average value of the brightness values ofall of the pixels included in the image may be used as the brightnessinformation about the image. When using the brightness information, itmay be determined that a scene change occurred between images when thebrightness information about a given image of the image sequence differsto a large extent from the brightness information about the image thatfollows the given image (e.g., when the difference in brightnessinformation is equal to or larger than a threshold value). When theimaging device includes a flash mechanism or the like, the brightnessinformation may change to a large extent by operating the mechanism evenif the object or the like has not changed. Therefore, it may bedesirable to use information other than the brightness information fordetecting a scene change, or use the brightness information andinformation other than the brightness information in combinationdepending on the configuration of the imaging device, for example.

Note that a scene change need not necessarily be detected based on themotion information, the imaging information about a specific object, thebrightness information, or the like. A scene change may be detectedusing various methods (e.g., chroma information (e.g., the degree ofredness when the image is an in vivo image)).

The above information for detecting a scene change may be used eitheralone or in combination. For example, when using the motion informationand the brightness information in combination, a determination may bemade based on whether or not a motion between images represented by themotion information is large, and whether or not a change in brightnessrepresented by the brightness information is large. In this case, ascene change may be detected when the motion and a change in brightnessare large, or may be detected when at least one of the motion and achange in brightness is large, for example.

The above configuration makes it possible to divide the image sequenceinto a plurality of partial image sequences based on a scene changedetected from the image sequence, and perform the deletion determinationprocess that utilizes the deformation information on each partial imagesequence. Since it is likely that the image that precedes the scenechange and the image that follows the scene change differ in the imagingtarget and the like, it is normally unnecessary to use such images forthe deletion determination process that utilizes the deformationinformation. Since the accuracy of the deletion determination processmay decrease as a result of calculating the deformation informationabout images that differ to a large extent, it may be desirable toperform the process on the image that precedes the scene change and theimage that follows the scene change. An efficient image summarizationprocess can be implemented by setting the partial image sequence basedon a scene change.

Note that the expression “precedes or follows the image” or “precedes orfollows the scene change” refers to the position within the imagesequence. Since it is considered that the image sequence is a set oftemporally or spatially continuous images, the position within the imagesequence can be defined based on the continuity. For example, an imageacquired at an earlier time precedes an image acquired at a later time.

The processing section 100 may detect a scene change based on accuracyinformation that represents the accuracy of the deformation informationused to determine whether or not the determination target image can bedeleted. Specifically, the processing section 100 may determine that ascene change has been detected when a value represented by the accuracyinformation is smaller than a given accuracy threshold value.

The accuracy information that represents the accuracy of the deformationinformation is calculated corresponding to each image or eachcombination of two images included in the image sequence. When theaccuracy represented by the accuracy information calculated from oneimage is low, the deformation information does not appropriatelyrepresent deformation between two images when the deformationinformation is calculated using the image and another image included inthe image sequence. Therefore, when calculating the accuracyinformation, the corresponding deformation information need notnecessarily be calculated in advance. The accuracy information may becalculated in advance, and whether or not to calculate the correspondingdeformation information may be determined based on the accuracyinformation. When the accuracy represented by the accuracy informationcalculated from a combination of two images is low, the deformationinformation does not appropriately represent deformation between the twoimages when the deformation information is calculated using the twoimages.

This makes it possible to detect a scene change based on the accuracyinformation that represents the accuracy of the deformation information.As illustrated in FIGS. 37A and 37B, when the accuracy of thedeformation information is low, it is difficult to sufficiently achievethe advantages of the process that utilizes the deformation information(i.e., preventing the occurrence of an area that cannot be observed, orpreventing a situation in which the attention area is missed). However,the accuracy of the deletion determination process can be improved bydetecting a scene change based on the accuracy information. Animprovement in accuracy may be achieved by utilizing the accuracyinformation independently of detection of a scene change. In this case,however, it is necessary to perform an additional process in order todetect a scene change. Therefore, the accuracy of the deletiondetermination process, and the efficiency and the speed of the imagesummarization process can be improved using a simple process byutilizing the accuracy information for detection of a scene change.

The processing section 100 may determine whether or not each pixelincluded in a selected image among the plurality of images is suitablefor calculation of the deformation information, calculate the accuracyinformation based on the number of pixels that have been determined tobe suitable for calculation of the deformation information, and detect ascene change at a position corresponding to the selected image withinthe image sequence based on the calculated accuracy information.

According to this configuration, when a given image has been selectedfrom the image sequence as the selected image, the accuracy informationabout the selected image can be calculated based on the determinationresult for each pixel of the selected image. The deformation informationmay be configured in various ways, and the deletion determinationprocess that utilizes the deformation information (deformation processin a narrow sense) may be implemented in various ways. The deformationinformation and the deletion determination process that utilizes thedeformation information are based on the fact that a given pixel withinone image corresponds to a pixel within the other image. Specifically,it is natural to determine the accuracy of the deformation informationbased on information on a pixel basis. In the eighth embodiment, whetheror not each pixel included in the selected image is suitable forcalculation of the deformation information is determined, and theaccuracy information is calculated from the determination results. Forexample, the ratio of the number of pixels (number of mask pixels) thatare suitable for calculation of the deformation information to the totalnumber of pixels of the selected image may be used as the accuracyinformation (see FIG. 38).

When it has been determined that the accuracy of the deformationinformation corresponding to the selected image is low based on theaccuracy information calculated from the selected image, it isdetermined that a scene change has been detected at the position of theselected image within the image sequence. For example, it may determinedthat a scene change occurred between the selected image and the imagethat immediately follows the selected image (see FIG. 35A). Since it isconsidered that the accuracy of the deformation information about theselected image and the image that immediately precedes the selectedimage is also low, it may determined that a scene change has beendetected between the selected image and the image that immediatelyfollows the selected image and between the selected image and the imagethat immediately precedes the selected image (see FIG. 35B).

The processing section 100 may set an area that has a given size andincludes the processing target pixel that is subjected to thedetermination as to suitability for calculation of the deformationinformation within the selected image, and determine whether or not theprocessing target pixel is suitable for calculation of the deformationinformation based on texture information about the set area.

The term “texture” refers to the pattern of a image. For example, atwo-dimensional Fourier transform or the like may be performed on theimage to calculate the spatial frequency power spectrum, and the spatialfrequency power spectrum may be used as the texture information. Notethat it suffices that the texture information is information thatrepresents the amount of pattern within the image. The textureinformation may include edge information that represents the contour ofthe object and the like.

This makes it possible to determine whether or not each pixel of theimage is suitable for calculation of the deformation information basedon the texture information. When a complex pattern is drawn, it isconsidered that the processing target pixel can be easily distinguishedfrom its peripheral pixel. Therefore, when the deformation informationabout the selected image and another image is calculated, it is likelythat the pixel included in the other image that corresponds to theprocessing target pixel is clear. When no pattern is present, it isdifficult to distinguish the processing target pixel from other pixels,and link the processing target pixel with the pixel included in theother image.

Note that whether or not each pixel is suitable for calculation of thedeformation information need not necessarily be determined using thetexture information. For example, the method disclosed inJP-A-2007-257287 or the like may be used.

The processing section 100 may calculate similarity information aboutthe similarity between a first image among the plurality of imagesincluded in the image sequence and a second image that immediatelyfollows the first image as the accuracy information. The processingsection 100 may detect a scene change at a position between the firstimage and the second image within the image sequence based on thecalculated accuracy information.

This makes it possible to use the similarity information about thesimilarity between two images (adjacent images) as the accuracyinformation. Specifically, it is considered that the deformationinformation can be accurately calculated when the similarity betweenimages is high. A known NCC or the like may be used as the similarityinformation

Alternatively, SSD or SAD that represents the degree of differencebetween images may be calculated, and the reciprocal thereof may be usedas the similarity information. In this case, the position of the scenechange may be the position between the first image and the second image(see FIG. 39).

When an ith (i is an integer) scene change and an (i+1)th scene changethat immediately follows the ith scene change have been detected fromthe image sequence, the processing section 100 may set images among theplurality of images included in the image sequence that follow the ithscene change and precede the (i+1)th scene change to be the partialimage sequence.

This makes it possible to set the partial image sequence based on thescene change (see FIG. 32B). Since the eighth embodiment is based on theassumption that a scene change is detected between images, for example,an image sequence that includes the image that immediately follows theith scene change as the starting point, and includes the image thatimmediately precedes the (i+1)th scene change as the end point, is setto be the partial image sequence. When only one image is present betweenthe ith scene change and the (i+1)th scene change, the image may beallowed to remain in the summary image sequence, or may be deleted (isdesirably allowed to remain in the summary image sequence taking accountof preventing the occurrence of an area that cannot be observed), andneed not be set to be the partial image sequence. Alternatively, theimage may be set to be the partial image sequence, and may be processed.In this case, since the reference image is set in the step S704, and thedetermination target image cannot be selected in the step S705, theimage is allowed to remain in the summary image sequence.

When a plurality of partial image sequences have been set, theprocessing section 100 may select the reference image and thedetermination target image from the plurality of partial image sequencesin parallel, and determine whether or not the determination target imagecan be deleted based on the deformation information about the referenceimage and the determination target image.

Specifically, when a jth (j is an integer) scene change has beendetected from the image sequence, the processing section 100 sets a kth(k is an integer) partial image sequence that includes images among theplurality of images included in the image sequence that precede the jthscene change, and a (k+1)th partial image sequence that includes imagesamong the plurality of images included in the image sequence that followthe jth scene change. In this case, a process that selects the referenceimage and the determination target image from the kth partial imagesequence, and determines whether or not the determination target imagecan be deleted based on the deformation information about the referenceimage and the determination target image, and a process that selects thereference image and the determination target image from the (k+1)thpartial image sequence, and determines whether or not the determinationtarget image can be deleted based on the deformation information aboutthe reference image and the determination target image, may be performedin parallel.

According to this configuration, since the deletion determinationprocess (S704 to S707 in FIG. 34) on each partial image sequence can beperformed in parallel, the speed of the image summarization process canbe increased.

The first to eighth embodiments according to the invention and themodifications thereof have been described above. Note that the inventionis not limited to the first to eighth embodiments and the modificationsthereof. Various modifications and variations may be made withoutdeparting from the scope of the invention. A plurality of elements amongthe elements described above in connection with the first to eighthembodiments and the modifications thereof may be appropriately combinedto implement various configurations. For example, an arbitrary elementmay be omitted from the elements described in connection with the firstto eighth embodiments and the modifications thereof. Arbitrary elementsamong the elements described above in connection with differentembodiments and/or modifications thereof may be appropriately combined.Any term cited with a different term having a broader meaning or thesame meaning at least once in the specification and the drawings can bereplaced by the different term in any place in the specification and thedrawings. Specifically, various modifications and applications arepossible without materially departing from the novel teachings andadvantages of the invention.

What is claimed is:
 1. An image processing device comprising: a storage;and a hardware processor that is configured to: acquire an imagesequence that includes a plurality of images; and perform an imagesummarization process that deletes some of the plurality of imagesincluded in the image sequence acquired by the image sequenceacquisition section to acquire a summary image sequence, whereinperforming the image summarization process comprises detecting a scenechange from the image sequence, setting a partial image sequence thatincludes images among the plurality of images included in the imagesequence based on the detected scene change, a position of the scenechange being used as a starting point or an end point of the partialimage sequence, selecting a reference image and a determination targetimage from the partial image sequence, and determining whether or notthe determination target image can be deleted based on deformationinformation about the reference image and the determination targetimage, and wherein the deformation information indicates a deformationparameter which represents how an object captured within the referenceimage is deformed within the determination target image, and theposition of the scene change corresponds to a position of an image inthe image sequence that has been determined not to be suitable forcalculation of the deformation information.
 2. A computer-readablestorage device with an executable program stored thereon, wherein theprogram is executable to control a computer to execute functionscomprising: acquiring an image sequence that includes a plurality ofimages; and performing an image summarization process that deletes someof the plurality of images included in the image sequence acquired bythe image sequence acquisition section to acquire a summary imagesequence, wherein performing the image summarization process comprisesdetecting a scene change from the image sequence, setting a partialimage sequence that includes images among the plurality of imagesincluded in the image sequence based on the detected scene change, aposition of the scene change being used as a starting point or an endpoint of the partial image sequence, selecting a reference image and adetermination target image from the partial image sequence, anddetermining whether or not the determination target image can be deletedbased on deformation information about the reference image and thedetermination target image, and wherein the deformation informationindicates a deformation parameter which represents how an objectcaptured within the reference image is deformed within the determinationtarget image, and the position of the scene change corresponds to aposition of an image in the image sequence that has been determined notto be suitable for calculation of the deformation information.
 3. Animage processing method comprising: acquiring an image sequence thatincludes a plurality of images; detecting a scene change from theacquired image sequence; setting a partial image sequence that includesimages among the plurality of images included in the image sequencebased on the detected scene change, a position of the scene change beingused as a starting point or an end point of the partial image sequence;selecting a reference image and a determination target image from thepartial image sequence; and performing an image summarization processthat determines whether or not the determination target image can bedeleted based on deformation information about the reference image andthe determination target image, and deletes some of the plurality ofimages included in the image sequence to acquire a summary imagesequence, wherein the deformation information indicates a deformationparameter which represents how an object captured within the referenceimage is deformed within the determination target image, and theposition of the scene change corresponds to a position of an image inthe image sequence that has been determined not to be suitable forcalculation of the deformation information.
 4. The image processingdevice according to claim 1, wherein the hardware processor is furtherconfigured to detect the scene change based on accuracy information thatrepresents an accuracy of the deformation information that is used todetermine whether or not the determination target image can be deleted.5. The image processing device according to claim 4, wherein thehardware processor is further configured to determine whether or noteach pixel included in a selected image among the plurality of images issuitable for calculation of the deformation information, calculate theaccuracy information based on a number of pixels that have beendetermined to be suitable for calculation of the deformationinformation, and detect the scene change at a position corresponding tothe selected image within the image sequence based on the calculatedaccuracy information.
 6. The image processing device according to claim5, wherein the hardware processor is further configured to set an areathat has a given size and includes a processing target pixel that issubjected to the determination as to suitability for calculation of thedeformation information within the selected image, and determine whetheror not the processing target pixel is suitable for calculation of thedeformation information based on texture information about the set area.7. The image processing device according to claim 4, wherein thehardware processor is further configured to calculate similarityinformation about a similarity between a first image among the pluralityof images and a second image that immediately follows the first image asthe accuracy information, and detect the scene change at a positionbetween the first image and the second image within the image sequencebased on the calculated accuracy information.
 8. The image processingdevice according to claim 4, wherein the hardware processor is furtherconfigured to determine that the scene change has been detected when avalue represented by the accuracy information is smaller than a givenaccuracy threshold value.
 9. The image processing device according toclaim 1, wherein the hardware processor is further configured to detectthe scene change based on at least one of motion information calculatedfrom the plurality of images, imaging information about a specificobject, and brightness information.
 10. The image processing deviceaccording to claim 1, wherein, when the hardware processor has detectedan ith (i is an integer) scene change and an (i+1)th scene change thatimmediately follows the ith scene change from the image sequence, thehardware processor is further configured to set images among theplurality of images included in the image sequence that follow the ithscene change and precede the (i+1)th scene change to be the partialimage sequence.
 11. The image processing device according to claim 1,wherein, when the hardware processor has set a plurality of the partialimage sequences, the hardware processor is further configured to selectthe reference image and the determination target image from theplurality of partial image sequences in parallel, and determine whetheror not the determination target image can be deleted based on thedeformation information about the reference image and the determinationtarget image.
 12. The image processing device according to claim 11,wherein, when the hardware processor has detected a jth (j is aninteger) scene change from the image sequence, the hardware processor isfurther configured to: set a kth (k is an integer) partial imagesequence that includes images among the plurality of images included inthe image sequence that precede the jth scene change, and a (k+1)thpartial image sequence that includes images among the plurality ofimages included in the image sequence that follow the jth scene change,and perform, in parallel, a process that selects the reference image andthe determination target image from the kth partial image sequence, anddetermines whether or not the determination target image can be deletedbased on the deformation information about the reference image and thedetermination target image, and a process that selects the referenceimage and the determination target image from the (k+1)th partial imagesequence, and determines whether or not the determination target imagecan be deleted based on the deformation information about the referenceimage and the determination target image.